Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.baseten.co/llms.txt

Use this file to discover all available pages before exploring further.

​
baseten_inference_requests_total

Cumulative number of requests to the model. Type: counter Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
status_code
label
required
The status code of the response.
​
is_async
label
required
Whether the request was an async inference request.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_end_to_end_response_time_seconds

End-to-end response time in seconds. Type: histogram Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
status_code
label
required
The status code of the response.
​
is_async
label
required
Whether the request was an async inference request.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_container_cpu_usage_seconds_total

Cumulative CPU time consumed by the container in core-seconds. Type: counter Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
replica
label
required
The ID of the replica.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_replicas_active

Number of replicas ready to serve model requests. Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_replicas_starting

Number of replicas starting up—that is, either waiting for resources to be available or loading the model. Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_container_restarts_total

Cumulative number of times the model container has been restarted. Restarts are typically caused by application crashes, out-of-memory kills, or failed liveness probes. See custom health checks for how liveness affects restart behavior. Type: counter
This metric rolls out behind a feature flag. Contact your account team if it’s not yet visible for your organization.
Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_pod_readiness

Number of pods grouped by their Kubernetes Ready condition. A pod with condition="true" is serving traffic; condition="false" means the pod is starting up, failing its readiness probe, or shutting down. Type: gauge
This metric rolls out behind a feature flag. Contact your account team if it’s not yet visible for your organization.
Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
condition
label
required
The Kubernetes Ready condition for the pods in this sample.Possible values:
  • "true": Pods are ready and serving traffic.
  • "false": Pods are starting up, failing readiness probes, or shutting down.
  • "unknown": The Ready condition can’t be determined (for example, the kubelet hasn’t reported recently).
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_container_cpu_memory_working_set_bytes

Working set memory usage of the container in bytes. Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
replica
label
required
The ID of the replica.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_request_size_bytes

Request size in bytes. Proxy for input tokens. Type: histogram Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
status_code
label
required
The status code of the response.
​
is_async
label
required
Whether the request was an async inference request.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_response_size_bytes

Response size in bytes. Proxy for generated tokens. Type: histogram Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
status_code
label
required
The status code of the response.
​
is_async
label
required
Whether the request was an async inference request.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_time_to_first_byte_seconds

Time to first byte/write in seconds. Proxy for time-to-first-token (TTFT). Type: histogram Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
status_code
label
required
The status code of the response.
​
is_async
label
required
Whether the request was an async inference request.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_time_in_async_queue_seconds

Time async requests spend queued before processing. Type: histogram Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_async_queue_size

Number of queued async requests over time. Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_gpu_memory_used

GPU memory used in MiB. Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
replica
label
required
The ID of the replica.
​
gpu
label
required
The ID of the GPU.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_gpu_utilization

GPU utilization as a percentage (between 0 and 100). Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
replica
label
required
The ID of the replica.
​
gpu
label
required
The ID of the GPU.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_ongoing_websocket_connections

Number of ongoing websocket connections. Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"

​
baseten_concurrent_requests

Number of in-progress concurrent inference requests for a deployment, including both requests currently being serviced by replicas and requests waiting in the queue. This is the primary signal that drives autoscaling decisions. Type: gauge Labels:
​
model_id
label
required
The ID of the model.
​
model_name
label
required
The name of the model.
​
deployment_id
label
required
The ID of the deployment.
​
environment
label
The environment that the deployment corresponds to. Empty if the deployment is not associated with an environment.
​
rollout_phase
label
The phase of the deployment in the promote to production process. Empty if the deployment is not associated with an environment.Possible values:
  • "promoting"
  • "stable"