from openai import OpenAI
import os

model_id = "abcd1234" # Replace with your model ID
deployment_id = "4321cbda" # [Optional] Replace with your deployment ID

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url=f"https://bridge.baseten.co/v1/direct"
)

response = client.chat.completions.create(
  model=f"meta-llama/Meta-Llama-3.1-70B-Instruct", # Replace with your model name
  messages=[
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ],
  extra_body={
    "baseten": {
      "model_id": model_id,
      "deployment_id": deployment_id
    }
  }
)

print(response.choices[0].message.content)

{
  "choices": [
      {
      "finish_reason": null,
      "index": 0,
      "message": {
          "content": "The 2020 World Series was played in Texas at Globe Life Field in Arlington.",
          "role": "assistant"
      }
      }
  ],
  "created": 1700584611,
  "id": "chatcmpl-eedbac8f-f68d-4769-a1a7-a1c550be8d08",
  "model": "abcd1234",
  "object": "chat.completion",
  "usage": {
      "completion_tokens": 0,
      "prompt_tokens": 0,
      "total_tokens": 0
  }
}

Use this endpoint with the OpenAI Python client and any deployment of a compatible model deployed on Baseten. If you’re serving a vLLM model in OpenAI compatible mode, this endpoint will support that model out of the box.

If your model does not have an OpenAI compatible mode, you can use the previous version of the bridge to make it compatible with OpenAI’s client, but with a more limited set of supported features.

​
Calling the model

https://bridge.baseten.co/v1/direct

​
Parameters

Parameters supported by the OpenAI ChatCompletions request can be found in the OpenAI documentation. Below are details about Baseten-specific arguments that must be passed into the bridge.

​
model
string
required

Typically Hugging Face repo name (e.g. meta-llama/Meta-Llama-3.1-70B-Instruct). In some cases, it may be another default specified by your inference engine.

​
extra_body
dict
required

Python dictionary that enables extra arguments to be supplied to the chat completion request.

​
extra_body.baseten
dict
required

Baseten-specific parameters that should be passed to the bridge. The arguments should be passed as a dictionary.

​
extra_body.baseten.model_id
string
required

The string identifier for the target model.

​
extra_body.baseten.deployment_id
string

The string identifier for the target deployment. When deployment_id is not provided, the production deployment will be used.

​
Output

Streaming and non-streaming responses are supported. The vLLM OpenAI Server is a good example of how to serve your model results.

For streaming outputs, data format must comply with the Server-Side-Events (SSE) format. A helpful example for JSON payloads can be found here.

​
Best Practices

  • Pin your openai package version in your requirements.txt file. This helps avoid any breaking changes that get introduced through package upgrades
  • If you must make breaking changes to your truss server (i.e. to introduce a new feature), you should first publish a new model deployment then update your API call on the client side.
from openai import OpenAI
import os

model_id = "abcd1234" # Replace with your model ID
deployment_id = "4321cbda" # [Optional] Replace with your deployment ID

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url=f"https://bridge.baseten.co/v1/direct"
)

response = client.chat.completions.create(
  model=f"meta-llama/Meta-Llama-3.1-70B-Instruct", # Replace with your model name
  messages=[
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ],
  extra_body={
    "baseten": {
      "model_id": model_id,
      "deployment_id": deployment_id
    }
  }
)

print(response.choices[0].message.content)

{
  "choices": [
      {
      "finish_reason": null,
      "index": 0,
      "message": {
          "content": "The 2020 World Series was played in Texas at Globe Life Field in Arlington.",
          "role": "assistant"
      }
      }
  ],
  "created": 1700584611,
  "id": "chatcmpl-eedbac8f-f68d-4769-a1a7-a1c550be8d08",
  "model": "abcd1234",
  "object": "chat.completion",
  "usage": {
      "completion_tokens": 0,
      "prompt_tokens": 0,
      "total_tokens": 0
  }
}

Was this page helpful?

Raise issue
Published deploymentChatCompletions (deprecated)