Metrics support matrix

When you build dashboards or alerts on Baseten metrics in your own observability stack, you need each metric’s exact name, type, and labels. This page is that reference: every metric the Baseten metrics export endpoint exposes. Baseten serves these metrics in Prometheus format at https://app.baseten.co/metrics. For the endpoint URL, authentication, scrape interval, and supported integrations (Prometheus, Datadog, Grafana, New Relic), see the export overview.

​

How to read this page

Each metric is listed by its Prometheus name, with:

• Type: counter (a cumulative total that only increases), gauge (a point-in-time value), or histogram (a distribution you can compute percentiles from).
• Labels: the dimensions you can filter and group by. Common labels are model_id, model_name, and deployment_id; environment and rollout_phase appear only for deployments tied to an environment. Some metrics, such as the engine metrics, use a smaller label set.

These are the same measurements shown in the dashboard Metrics tab, exposed here for export. The two sets overlap but aren’t identical: a few values are graphed in the dashboard without being exportable, and some exported metrics aren’t graphed.

​

Availability

Some metrics are emitted only for certain deployments:

• vLLM and SGLang metrics appear when Baseten detects that engine on your deployment. See vLLM and SGLang metrics.
• Pod health metrics (baseten_container_restarts_total and baseten_pod_readiness) roll out behind a feature flag. Contact your account team if they aren’t yet visible for your organization.

---

​

baseten_inference_requests_total

Cumulative number of requests to the model. Type: counter Labels:

​

baseten_end_to_end_response_time_seconds

End-to-end response time in seconds. Type: histogram Labels:

​

baseten_container_cpu_usage_seconds_total

Cumulative CPU time consumed by the container in core-seconds. Type: counter Labels:

​

baseten_replicas_active

Number of replicas ready to serve model requests. Type: gauge Labels:

​

baseten_replicas_starting

Number of replicas starting up—that is, either waiting for resources to be available or loading the model. Type: gauge Labels:

​

baseten_container_restarts_total

Cumulative number of times the model container has been restarted. Restarts are typically caused by application crashes, out-of-memory kills, or failed liveness probes. See custom health checks for how liveness affects restart behavior. Type: counter Labels:

​

baseten_pod_readiness

Number of pods grouped by their Kubernetes Ready condition. A pod with condition="true" is serving traffic; condition="false" means the pod is starting up, failing its readiness probe, or shutting down. Type: gauge Labels:

​

baseten_container_cpu_memory_working_set_bytes

Working set memory usage of the container in bytes. Type: gauge Labels:

​

baseten_request_size_bytes

Request size in bytes. Proxy for input tokens. Type: histogram Labels:

​

baseten_response_size_bytes

Response size in bytes. Proxy for generated tokens. Type: histogram Labels:

​

baseten_time_to_first_byte_seconds

Time to first byte/write in seconds. Proxy for time-to-first-token (TTFT). Type: histogram Labels:

​

baseten_time_in_async_queue_seconds

Time async requests spend queued before processing. Type: histogram Labels:

​

baseten_async_queue_size

Number of queued async requests over time. Type: gauge Labels:

​

baseten_async_webhook_requests_total

Cumulative number of async inference webhook delivery requests sent. Type: counter Labels:

​

baseten_async_webhook_latency_seconds

Latency of async inference webhook delivery requests in seconds. Type: histogram Labels:

​

baseten_gpu_memory_used

GPU memory used in MiB. Type: gauge Labels:

​

baseten_gpu_utilization

GPU utilization as a percentage (between 0 and 100). Type: gauge Labels:

​

baseten_ongoing_websocket_connections

Number of ongoing websocket connections. Type: gauge Labels:

​

baseten_concurrent_requests

Total in-flight inference requests for a deployment, including both requests currently being serviced by replicas and requests waiting to be processed. Async inference requests are not included in this metric. This is the primary signal that drives autoscaling decisions. Type: gauge Labels:

​

vLLM and SGLang metrics

When Baseten detects vLLM or SGLang on your deployment, it scrapes your container’s /metrics endpoint and exports the engine’s native metrics alongside Baseten’s own. These also appear as graphs in the Metrics tab. The engines define these metrics, not Baseten, and they change between versions. For the complete, current list, always refer to the official vLLM and SGLang metrics documentation. Baseten normalizes these metrics across engine versions and exports the most useful ones. Some exported metrics include tokens per second, time to first token, KV cache usage, and the number of requests running or queued. Baseten attaches the same two labels to every exported engine metric: