Deploy a model

Deploy Gemma 4 E2B Instruct through Explore, the Baseten CLI, or the Management API. Each path creates a dedicated endpoint that you can call from Python. You’ll need a Baseten account. This example uses an L4 GPU. Review pricing before deploying; provisioned replicas incur charges even while idle.

Deploy the model

Follow one tab. Each creates a separate deployment of the same model. Explore supplies a suggested configuration; the CLI and API use the Gemma recipe, so serving settings can differ.

Baseten CLI
Dashboard
Management API

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten

Download and extract the Windows archive for the latest Baseten CLI release:

Terminal

$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .

Move baseten.exe to a directory on your PATH.

For other platforms or a specific version, see the Baseten CLI install reference.Sign in:

Terminal

baseten auth login

Prepare the configuration

Create a project directory:

Terminal

mkdir gemma-4-e2b
cd gemma-4-e2b

Copy config.yaml from the E2B tab of the Gemma 4 recipe into this directory. It defines the weights, vLLM server, and L4 hardware.Accept the model’s Hugging Face terms. Save a read token with access to its weights in Baseten secrets as hf_access_token, the name used by the recipe.

Push the configuration

From the project directory, run:

baseten model push --tail --wait

  Model:       model:gemma-4-E2B-it preset:latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

--tail shows logs; --wait waits for Active. Save your model ID and continue to call the model.

Open Explore, search for Gemma, and select Gemma 4 E2B Instruct.
Under Deployment options, select Configure deployment on the Dedicated deployment card.
Review the suggested L4 GPU, price per minute, and autoscaling settings.
Select Deploy. The dashboard shows the deployment’s progress.

Continue to wait for the deployment.

Prepare the configuration

Prepare a directory containing the E2B config.yaml from the Gemma 4 recipe. Accept the model’s Hugging Face access terms and save a read token with model access in Baseten secrets as hf_access_token. You don’t need to install either CLI.Create a personal key in API keys and set it in your terminal:

export BASETEN_API_KEY="YOUR_API_KEY"

$env:BASETEN_API_KEY = "YOUR_API_KEY"

Create the deployment

Save this as deploy.py beside config.yaml. It prepares an upload, uploads the configuration using Baseten’s temporary S3 credentials, and creates the model. The configuration tells Baseten where to download the weights.

deploy.py

import io
import os
from pathlib import Path
import tarfile

import boto3
import requests
import yaml

raw_config = Path("config.yaml").read_text()
config = yaml.safe_load(raw_config)
deployment = {
    "config": config,
    "raw_config": raw_config,
    "environment_name": "production",
}

api = requests.Session()
api.headers["Authorization"] = f"Bearer {os.environ['BASETEN_API_KEY']}"

# Validate the configuration and get temporary upload credentials.
prepared = api.post(
    "https://api.baseten.co/v1/prepare_model_upload",
    json={"name": config["model_name"], "deployment": deployment},
    timeout=60,
)
prepared.raise_for_status()
upload = prepared.json()

# Package this config-only project, with config.yaml at the archive root.
archive = io.BytesIO()
with tarfile.open(fileobj=archive, mode="w:gz") as tar:
    tar.add("config.yaml", arcname="config.yaml")
archive.seek(0)

creds = upload["creds"]
s3 = boto3.client(
    "s3",
    aws_access_key_id=creds["aws_access_key_id"],
    aws_secret_access_key=creds["aws_secret_access_key"],
    aws_session_token=creds["aws_session_token"],
    region_name=upload["s3_region"],
)
s3.upload_fileobj(archive, upload["s3_bucket"], upload["s3_key"])

# Create the model only after the upload succeeds.
created = api.post(
    "https://api.baseten.co/v1/models",
    json={
        "source": {
            "kind": "model_archive",
            "name": config["model_name"],
            "s3_key": upload["s3_key"],
            "deployment": deployment,
        }
    },
    timeout=60,
)
created.raise_for_status()
result = created.json()
print("Model ID:", result["model"]["id"])
print("Deployment ID:", result["deployment"]["id"])
print("Status:", result["deployment"]["status"])

Run it with uv, which installs the script’s dependencies for this run:

Terminal

uv run --with requests --with boto3 --with pyyaml deploy.py

Save the IDs the script prints, then wait for the deployment. Rerunning the script creates another model.This script packages only config.yaml. Include any custom code or bundled files when adapting it to another project.To deploy a model published through Baseten for Model Labs, see Deploy a Model Labs listing.

Deploy to a specific team

The script creates the model in the organization’s default team. To choose a team, get its ID from List teams. Add "team_id": "<team_id>" to the prepare-upload request body, alongside name and deployment, and send the create request to POST /v1/teams/{team_id}/models with the same body as the script’s create request. Use the same team ID in both requests.

Wait for the deployment

Open Dedicated Inference in the dashboard, select your model, and wait for Active. The CLI’s --wait flag already waits for this state. If startup fails, check the deployment’s build and runtime logs. For API automation, poll Get deployment with the model and deployment IDs until status is ACTIVE.

Call the model

Use the same OpenAI-compatible request for any deployment path. Find your model ID in the CLI or API output, or select Copy ID beside the model name in the dashboard. Create a personal key in API keys if you don’t have one. Set it in the terminal where you’ll run the client:

export BASETEN_API_KEY="YOUR_API_KEY"

$env:BASETEN_API_KEY = "YOUR_API_KEY"

Save this as chat.py and replace {model_id} with your model ID:

chat.py

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
    api_key=os.environ["BASETEN_API_KEY"],
)

response = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {"role": "user", "content": "Explain inference in one sentence."}
    ],
)

print(response.choices[0].message.content)

Run it with uv. The response wording varies:

uv run --with openai chat.py

Inference is the process of using a trained model to generate an output from new input.

The URL selects your deployed model’s production environment. The model field matches the serving name in its configuration.

Clean up

Exiting the client script doesn’t stop the deployment. When you’re finished testing, deactivate the deployment to release its replicas and stop its compute charges. Its endpoint stops serving requests; the model and configuration remain available for reactivation.

Open Dedicated Inference in the dashboard and select the model you created.
Under Deployments, select the deployment you used in this example.
Select Deactivate deployment, then Yes, deactivate.
Wait for the deployment’s status to become Inactive.

See Manage the deployment lifecycle to reactivate it later.

Next steps

Configure autoscaling to control replica counts and scale to zero when idle.
Configure a model server to learn how to write your own Truss configuration.
Add a deployment to publish another version under the existing model ID.

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Baseten for Model Labs

Deploy the model

Prepare the configuration

Push the configuration

Prepare the configuration

Create the deployment

Wait for the deployment

Call the model

Clean up

Next steps

​Deploy the model

​Install and sign in

​Prepare the configuration

​Push the configuration

​Prepare the configuration

​Create the deployment

​Wait for the deployment

​Call the model

​Clean up

​Next steps

Deploy the model

Install and sign in

Prepare the configuration

Push the configuration

Prepare the configuration

Create the deployment

Wait for the deployment

Call the model

Clean up

Next steps