Customize your view
By default the Metrics tab shows a standard set of graphs. Use the Customize view button at the top of the tab to show, hide, and reorder any graph, and your layout is saved per model. A hidden graph stays in the Customize view panel, so you can turn it back on at any time.Events
Turn on the Events toggle at the top of the Metrics tab to overlay platform events on your graphs. When response time jumps or replica count changes, a marker shows whether a deployment, promotion, or settings change caused it. Events are available for models, not for shared Model API endpoints or training jobs. The toggle is off by default. Baseten marks these events:- Deployed: a new deployment, with its target environment.
- Promoted: a deployment promoted to an environment.
- Promotion control action: a pause, resume, or roll-forward during a promotion.
- Autoscaling changed: a new replica range or concurrency target.
- Activated and Deactivated: a deployment turned on or off.
- Instance type changed: a move to a new instance type.
- Replica terminated: an individual replica shut down.
- Environment updated: a change to an environment.
Inference volume
Tracks the response rate over time, segmented by HTTP status codes:2xx: 🟢 Successful requests4xx: 🟡 Client errors5xx: 🔴 Server errors (includes model prediction exceptions)
For non-HTTP models and Chains (WebSockets and gRPC), the status codes reflect the status codes for those protocols. For a full list of the WebSocket close codes surfaced here, see WebSocket status codes.
Response time
Measured at different percentiles (p50, p90, p95, p99):- End-to-end response time: Includes cold starts, queuing, and inference (excludes client-side latency). Reflects real-world performance.
- Inference time: Covers only model execution, including pre/post-processing. Useful for optimizing single-replica performance.
- Time to first byte: Measures the time from request receipt to the first byte Baseten writes back, including any queueing and routing time. It approximates time to first token (TTFT), because Baseten doesn’t parse tokens out of your model’s response. Model API endpoints graph time to first token instead.
Request and response size
Measured at different percentiles (p50, p90, p95, p99):- Request size: Tracks the request size distribution. A proxy for input tokens.
- Response size: Tracks the response size distribution. A proxy for generated tokens.
Replicas
Tracks the number of active and starting replicas:- Starting: Waiting for resources or loading the model.
- Active: Ready to serve requests.
- For development deployments, a replica is considered active while running the live reload server.
baseten_pod_readiness.
Restarts
Tracks the cumulative number of times the model container has been restarted. Restarts are typically caused by application crashes, out-of-memory kills, or failed liveness probes. Frequent restarts usually indicate one of:- A crash in
load()or in your model code. - An out-of-memory event: check the Memory usage graph.
- A liveness probe failing under load: review
restart_threshold_secondsand any custom health check logic.
Concurrent requests
Total in-flight inference requests across replicas, including both requests currently being serviced and requests waiting to be processed. Async inference requests are not included in this metric. This is the primary signal that drives autoscaling decisions. For the full metric definition and labels, seebaseten_concurrent_requests.
This metric is a point-in-time gauge, sampled roughly every 30 seconds, while inference volume counts every request over the full minute. The two relate through Little’s Law:
average concurrency ≈ requests per second × average end-to-end latency
When requests are fast, that product stays well below 1 even at high volume, so most samples catch the system empty and the gauge reads 0. For example, 600 requests per minute at 80 ms latency averages about 0.8 requests in flight. Autoscaling still responds correctly, because it acts on sustained concurrency rather than sub-second bursts.
CPU usage and memory
Displays resource utilization across replicas. Metrics are averaged and may not capture short spikes.Considerations:
- High CPU/memory usage: May degrade performance. Consider upgrading to a larger instance type.
- Low CPU/memory usage: Possible overprovisioning. Switch to a smaller instance to reduce costs.
GPU usage and memory
Shows GPU utilization across replicas.- GPU usage: Percentage of time a kernel function occupies the GPU.
- GPU memory: Total memory used.
Considerations:
- High GPU load: Can slow inference. Check response time metrics.
- High memory usage: May cause out-of-memory failures.
- Low utilization: May indicate overprovisioning. Consider a smaller GPU.
Organization GPU usage
The GPU graphs above cover a single deployment. To see how many GPUs your whole workspace is using at once, organization admins can open the GPU usage tab in Organization settings. This view counts active GPUs (ready replicas multiplied by the GPUs each replica uses), aggregated across every model and deployment in the organization. To view organization GPU usage:- Sign in to your workspace at app.baseten.co and open Organization settings.
- Choose the GPU usage tab.
vLLM and SGLang metrics
When your deployment serves an LLM with vLLM or SGLang, Baseten surfaces engine-native metrics in the Metrics tab alongside the standard ones. These graphs report what the inference engine itself measures: metrics like tokens per second, time to first token, KV cache usage, and the number of requests running or queued.How detection works
You don’t turn these graphs on manually. Baseten scrapes your container’s/metrics endpoint and looks for metrics that match the vLLM or SGLang format. When it finds them, the matching graphs appear in the Metrics tab automatically. No configuration or redeploy is required.
If you don’t see the graphs, and they don’t appear in the Customize view panel either, Baseten was most likely unable to read your container’s metrics endpoint. Common causes are that the endpoint isn’t exposed, it’s blocking Baseten’s scrape, or the engine isn’t emitting metrics yet. Confirm that your engine serves Prometheus metrics on its /metrics route. For custom servers, routes like /metrics pass through to your server unchanged.
Detection runs on a periodic scrape and results are cached, so a deployment that just started exporting metrics may take a few minutes to show its graphs.
Show and hide graphs
Many of these engine graphs are hidden by default. Turn them on with Customize your view. The exact graphs depend on what your engine version emits. The latency graphs are shown at the p50, p90, p95, and p99 percentiles, and counters are summed over the selected time range.Export engine metrics
The Metrics tab shows a curated set of graphs. You can also export the underlying vLLM and SGLang metrics, along with a few that aren’t graphed in the dashboard, to your own observability stack through the metrics export endpoint. See vLLM and SGLang metrics for the labels Baseten adds.BIS-LLM metrics
When your deployment runs on the BIS-LLM engine, the Metrics tab adds engine graphs alongside the standard metrics above. They appear automatically for BIS-LLM deployments; no configuration is required.- Token throughput: input and output tokens over time, with per-request distributions for input tokens, output tokens, and tokens per second.
- KV cache hit rate: the cache hit rates your workers observe. Falling hit rates mean more prefill work per request and higher latency.
- Time to first byte: on BIS-LLM deployments this graph is measured inside the engine, covering engine queueing and prefill but not platform routing, unlike the standard time to first byte, which includes routing time. Response time still covers the end-to-end view.
- Inflight tokens per worker: the load signal the BIS-LLM autoscaler acts on, covering tokens being processed and queued at the router.
Export engine metrics
The underlying series carry thebaseten_llm_* prefix and export to your own observability stack through the metrics export endpoint. That includes the raw accepted and draft token counters behind the acceptance rate graph, so you can rebuild it in your own dashboards. For the full list with types and labels, see BIS-LLM metrics.
Model APIs metrics
Model API endpoints have their own Metrics tab. Because you call a shared endpoint rather than run your own replicas, the tab graphs request and token metrics, not the replica and hardware metrics above.- Inference volume: response rate over time, segmented by HTTP status code.
- Rate limit: rate of requests the endpoint rejects for exceeding your rate limits and budgets.
- End-to-end response time: time from request receipt to the last byte of the response, including queueing and inference.
- Time to first token: time from request receipt to the first generated token, including queueing and routing. The inference engine reports this timing directly, unlike time to first byte on your own deployments.
- Request size and Response size: request and response payload size distributions.
- Tokens usage: tokens per second the endpoint processes for your workspace.
Async queue metrics
- Time in Async Queue: Time spent in the async queue before execution (p50, p90, p95, p99).
- Async Queue Size: Number of queued async requests.
- Webhook requests: Number of async webhook delivery requests sent.
- Webhook latency: Latency of async webhook delivery requests (p50, p90, p95, p99).
Considerations:
- Large queue size indicates requests are queued faster than they are processed.
- To improve async throughput, increase the max replicas or adjust autoscaling concurrency.
- Async Queue Size is a point-in-time gauge, like concurrent requests. When requests spend little time queued, most samples catch an empty queue and it reads 0 even under steady load.
Use metrics for autoscaling
Use these metrics to diagnose autoscaling behavior and tune your settings.Key metrics to watch
Diagnose autoscaling issues
For solutions to common autoscaling problems, see Autoscaling troubleshooting.