Model I/O in binary

Send bytes to a custom Truss model with MessagePack, or read raw bytes from a streaming response. The client and model must agree on the payload format. You need a Baseten API key in BASETEN_API_KEY and the Baseten CLI and Truss installed. Install the client dependencies with pip install requests msgpack.

Deploy a basic Truss for binary I/O

If you need a deployed model to try the invocation examples below, follow these steps to create and deploy a minimal Truss that accepts and returns binary data. The Truss performs no operations and is purely illustrative.

Steps for deploying an example Truss

To create and deploy the example Truss:

Create a Truss:
Terminal
truss init binary_test
This creates a Truss in a new directory binary_test. By default, newly created Trusses implement an identity function that returns the exact input they are given.

Optionally, modify binary_test/model/model.py to log that the data received is of type bytes:

binary_test/model/model.py

def predict(self, model_input):
    # Run model inference here
    print(f"Input type: {type(model_input['byte_data'])}")
    return model_input

Deploy the Truss to Baseten. Keep the watch process running, wait for the deployment to become active, and use another terminal for the client requests:
Terminal
baseten model push --dir binary_test --watch

Send raw bytes as model input

To send bytes to your model: Configure the request as follows:

Set the content-type HTTP header to application/octet-stream.
Use msgpack to encode the data or file.
Make a POST request to the model.

Set BASETEN_MODEL_ID to the identity Truss model ID. Run the following script to send a short byte string and check that the returned bytes match:

import os
import requests
import msgpack

model_id = os.environ["BASETEN_MODEL_ID"]
source = "Café from Baseten.".encode("utf-8")
response = requests.post(
    f"https://model-{model_id}.api.baseten.co/development/predict",
    headers={
        "Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}",
        "Content-Type": "application/octet-stream",
        "Accept": "application/octet-stream",
    },
    data=msgpack.packb({"byte_data": source}),
    timeout=60,
)
response.raise_for_status()
output = msgpack.unpackb(response.content)
print("Bytes match:", output["byte_data"] == source)

Bytes match: True

The development URL matches the --watch deployment above. For a published production deployment, use /environments/production/predict.

To support certain types like numpy and datetime values, you may need to extend client-side msgpack encoding with the same encoder and decoder used by Truss.

Parse raw bytes from model output

To save the returned bytes: Continue the previous script and write the decoded byte field to a file:

from pathlib import Path

Path("output.bin").write_bytes(output["byte_data"])
print(Path("output.bin").read_bytes().decode("utf-8"))

Café from Baseten.

MessagePack decoding returns the payload your model produced. Use the appropriate field and extension for your model’s output.

Streaming binary outputs

Streaming sends raw response chunks rather than one MessagePack object. Decode them according to the format your model yields. To stream bytes from a Truss:

Replace the model’s predict method with a generator and let the watch process update the deployment:

model/model.py

class Model:
    def predict(self, model_input):
        yield model_input["text"].encode("utf-8")

Run the following client to save chunks directly and read the completed file:

import os
from pathlib import Path
import requests

with requests.post(
    f"https://model-{os.environ['BASETEN_MODEL_ID']}.api.baseten.co/development/predict",
    headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
    json={"text": "Café from Baseten."},
    stream=True,
    timeout=60,
) as response:
    response.raise_for_status()
    with open("output.bin", "wb") as output:
        for chunk in response.iter_content(chunk_size=None):
            output.write(chunk)
print(Path("output.bin").read_text(encoding="utf-8"))

Café from Baseten.

For application-specific completion records and interrupted streams, see Streaming.

Get started

Model APIs

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Model I/O in binary

Deploy a basic Truss for binary I/O

Send raw bytes as model input

Parse raw bytes from model output

Streaming binary outputs

Next steps

File I/O

Streaming

​Deploy a basic Truss for binary I/O

​Send raw bytes as model input

​Parse raw bytes from model output

​Streaming binary outputs

​Next steps

File I/O

Streaming

Deploy a basic Truss for binary I/O

Send raw bytes as model input

Parse raw bytes from model output

Streaming binary outputs

Next steps