---
name: Baseten
description: Use when deploying machine learning models to production, calling hosted LLMs through APIs, training models on GPU infrastructure, or building multi-step inference pipelines. Reach for Baseten when you need to manage model inference endpoints, configure autoscaling, handle deployment environments, or integrate with observability systems.
metadata:
    mintlify-proj: baseten
    version: "1.0"
---

# Baseten Skill

## Product summary

Baseten is a platform for model inference and training on production GPU infrastructure. Deploy custom models with `config.yaml` and the Baseten CLI, call hosted LLMs through OpenAI-compatible APIs (Model APIs), or train models and serve checkpoints. Baseten manages containers, GPU capacity across clouds and regions, autoscaling, and observability.

**Key files and commands:**
- `config.yaml`: Defines model dependencies, resources, environment variables, and serving configuration
- `model/model.py`: Custom Python logic for preprocessing, inference, and postprocessing
- `baseten model push`: Deploy a model to production
- `baseten model push --watch`: Deploy with live code patching for rapid iteration
- `baseten model predict`: Call a deployed model from the CLI
- Primary docs: https://docs.baseten.co

## When to use

Reach for Baseten when:
- **Deploying models**: You have a Hugging Face model, fine-tuned checkpoint, or custom model that needs a production endpoint
- **Calling hosted LLMs**: You want to use DeepSeek, GLM, Kimi, or other supported models through an OpenAI-compatible API without managing infrastructure
- **Iterating locally**: You're developing a model and need live code patching without full redeeploys
- **Managing environments**: You need separate development, staging, and production deployments with stable endpoints
- **Scaling inference**: You need autoscaling based on traffic, concurrency targets, or schedules
- **Building pipelines**: You're composing multiple models or services into a single workflow (use Chains)
- **Training models**: You're fine-tuning with Loops or running custom training code with Training Jobs
- **Monitoring production**: You need logs, metrics, and observability exports to Datadog, Prometheus, or other systems

## Quick reference

### Deployment workflow

| Task | Command |
| --- | --- |
| Create a model directory | `mkdir my-model && cd my-model` |
| Initialize a model | `baseten model init` (or write `config.yaml` manually) |
| Deploy to production | `baseten model push` |
| Deploy with live patching | `baseten model push --watch` |
| Re-attach to watch session | `baseten model watch` |
| Call a model | `baseten model predict --model-id <id>` |
| List models | `baseten model list` |
| View deployment logs | `baseten model deployment logs --model-id <id> --deployment-id <id>` |
| Promote to production | `baseten model deployment promote --model-id <id> --deployment-id <id>` |

### config.yaml essentials

```yaml
model_name: my-model
resources:
  accelerator: L4  # or H100, A100, etc.
  use_gpu: true
requirements:
  - torch==2.0.0
  - transformers==4.30.0
environment_variables:
  MY_VAR: value
secrets:
  api_key: null  # Set in Baseten dashboard
runtime:
  predict_concurrency: 32  # Concurrent requests per replica
```

### Model class structure

```python
class Model:
    def __init__(self, **kwargs):
        # Read config, data_dir, secrets, environment
        self._config = kwargs.get("config")
        self._secrets = kwargs.get("secrets")
    
    def load(self):
        # Download weights, initialize model (runs once at startup)
        pass
    
    def predict(self, model_input):
        # Run inference on each request
        return {"output": "result"}
```

### Inference endpoints

| Target | URL pattern |
| --- | --- |
| Production environment | `https://model-{model_id}.api.baseten.co/environments/production/predict` |
| Development deployment | `https://model-{model_id}.api.baseten.co/development/predict` |
| Specific deployment | `https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/predict` |
| OpenAI-compatible (vLLM) | `https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions` |

### Authentication

```bash
# Set API key
export BASETEN_API_KEY="your-api-key"

# Or use Bearer token in requests
curl -H "Authorization: Bearer $BASETEN_API_KEY" https://...
```

### Autoscaling config

```yaml
# In dashboard or via API
min_replica: 0              # Scale to zero when idle
max_replica: 10             # Maximum replicas
concurrency_target: 32      # Requests per replica before scaling
target_utilization_percentage: 70  # Headroom before scale-up
scale_down_delay: 900       # Seconds to wait before removing replicas
```

## Decision guidance

### When to use config.yaml vs. Model class

| Scenario | Use config.yaml | Use Model class |
| --- | --- | --- |
| Deploying open-source Hugging Face model | ✓ | |
| Custom preprocessing/postprocessing | | ✓ |
| Multiple models in one endpoint | | ✓ |
| Unsupported model architecture | | ✓ |
| Simple vLLM or SGLang server | ✓ | |

### When to use development vs. production deployments

| Scenario | Development | Production |
| --- | --- | --- |
| Iterating on code | ✓ | |
| Testing before release | ✓ | |
| Serving real traffic | | ✓ |
| Single replica only | ✓ | |
| Full autoscaling | | ✓ |
| Live code patching | ✓ | |

### When to use Model APIs vs. dedicated deployments

| Scenario | Model APIs | Dedicated |
| --- | --- | --- |
| Using supported hosted model (DeepSeek, GLM, etc.) | ✓ | |
| Custom model or fine-tune | | ✓ |
| Pay-per-token pricing | ✓ | |
| Fixed infrastructure cost | | ✓ |
| No deployment needed | ✓ | |
| Full control over hardware | | ✓ |

## Workflow

### Deploy a model for the first time

1. **Create project structure**: Make a directory with `config.yaml` (or `config.yaml` + `model/model.py` for custom logic)
2. **Define resources**: Specify GPU type, Python packages, environment variables in `config.yaml`
3. **Authenticate**: Run `baseten auth login --web` or set `BASETEN_API_KEY`
4. **Push to production**: Run `baseten model push` from the project directory
5. **Wait for deployment**: Monitor logs with `baseten model deployment logs --model-id <id> --deployment-id <id>`
6. **Test the endpoint**: Call with `baseten model predict --model-id <id>` or cURL with your API key
7. **Promote if needed**: Use `baseten model deployment promote` to move to a named environment

### Iterate on a model locally

1. **Start watch mode**: Run `baseten model push --watch` to create a development deployment and watch for changes
2. **Edit code**: Modify `model.py`, `config.yaml`, or requirements
3. **Save and test**: Changes auto-patch in seconds; test with `baseten model predict --model-id <id>`
4. **Repeat**: Edit, save, test until ready for production
5. **Deploy to production**: Stop watch (Ctrl+C), then run `baseten model push` to create a published deployment
6. **Promote**: Use `baseten model deployment promote` to move to production environment

### Call a deployed model

1. **Get model ID**: From dashboard URL or `baseten model list`
2. **Create API key**: Generate at https://app.baseten.co/settings/api_keys
3. **Set environment**: `export BASETEN_API_KEY="your-key"`
4. **Make request**: Use cURL, Python requests, or OpenAI SDK with the model's endpoint URL
5. **Handle response**: Parse JSON response or stream tokens for LLM endpoints

### Scale and monitor

1. **Check current settings**: `baseten model environment describe --model-id <id> --environment production`
2. **Update autoscaling**: `baseten model deployment update-autoscaling --model-id <id> --deployment-id <id> --min-replica 1 --max-replica 10`
3. **View metrics**: `baseten model deployment metrics --model-id <id> --deployment-id <id>`
4. **Stream logs**: `baseten model deployment logs --model-id <id> --deployment-id <id> --follow`
5. **Export metrics**: Configure Prometheus, Datadog, or Grafana scrape in dashboard

## Common gotchas

- **API key not set**: Requests fail with 401. Export `BASETEN_API_KEY` or pass `Authorization: Bearer <key>` header.
- **Model not ready**: Deployment shows "LOADING_MODEL" or "BUILDING". Wait for "ACTIVE" status before calling.
- **Cold start latency**: First request after scale-to-zero is slow. Set `min_replica: 1` to keep warm, or use `baseten model wake` to pre-warm.
- **Concurrency target too low**: Replicas scale up unnecessarily. Benchmark your model and set concurrency_target to actual throughput capacity (e.g., 32–128 for vLLM).
- **Concurrency target too high**: Requests queue at replicas instead of scaling. Lower concurrency_target or increase max_replica.
- **Live patch fails silently**: Some changes (GPU type, system packages, data directory) require full redeploy. Stop watch and run `baseten model push` again.
- **Hot reload doesn't re-run `__init__` or `load`**: New instance state added in those methods won't be visible in `predict`. Do a full reload with `baseten model push --watch`.
- **Secrets not injected**: Secrets defined in `config.yaml` must be set in Baseten dashboard first. They're `None` at runtime if not configured.
- **Model weights too large**: Bundling >1 GB in `data/` slows every cold start. Use Baseten Delivery Network (BDN) instead with `weights` in config.yaml.
- **Async requests not completing**: Check webhook configuration and ensure your callback URL is reachable. Use `baseten model deployment logs` to debug.
- **Regional endpoint 403**: Deployment is regional but you're calling non-regional endpoint. Use the regional endpoint URL instead.

## Verification checklist

Before submitting a deployment:

- [ ] `config.yaml` is valid YAML with required fields (`model_name`, `resources`)
- [ ] Python requirements are pinned to specific versions (e.g., `torch==2.0.0`)
- [ ] Secrets are defined in `config.yaml` but values are set in Baseten dashboard
- [ ] Model loads within 30-minute timeout (check logs if deployment fails)
- [ ] Endpoint responds to a test request with correct input/output format
- [ ] Autoscaling settings match your traffic pattern (concurrency_target, min/max replicas)
- [ ] Logs are accessible via `baseten model deployment logs`
- [ ] Metrics are visible in dashboard or exported to observability system
- [ ] Environment promotion is configured (development → staging → production)
- [ ] API key has access to the model (check in dashboard)

## Resources

- **Comprehensive page listing**: https://docs.baseten.co/llms.txt
- **Build your first model**: https://docs.baseten.co/development/model/build-your-first-model
- **How Baseten works**: https://docs.baseten.co/concepts/howbasetenworks
- **Deployment environments**: https://docs.baseten.co/deployment/environments
- **Autoscaling**: https://docs.baseten.co/deployment/autoscaling/overview
- **Baseten CLI reference**: https://docs.baseten.co/reference/cli/baseten/overview
- **Inference API reference**: https://docs.baseten.co/reference/inference-api/overview
- **Management API reference**: https://docs.baseten.co/reference/management-api/overview

---

> For additional documentation and navigation, see: https://docs.baseten.co/llms.txt