SamplingClient generates text completions from a base model or published LoRA weights. ServiceClient.create_sampling_client creates a standalone sampler that serves a base model or a fixed checkpoint. TrainingClient.save_weights_and_get_sampling_client returns a client on the run’s paired sampler: serverless clients stay pinned to the published checkpoint, and clients in the dedicated mode serve that version or a later one. Both clients expose the same sample method.
For sampler deployments, the SDK configures a large HTTP connection pool by default. Override it with BT_LOOPS_MAX_CONNECTIONS when you need a different limit, for example on a VM behind Cloud NAT with a tight port budget. Serverless samplers multiplex requests over HTTP/2.
Generate from the current weights:
SampleResponse
Generate
num_samples completions from prompt (a ModelInput). Pass a SamplingParams instance to control temperature, top-p, top-k, max tokens, seed, stop sequences, and structured outputs; omit it to use defaults. Set include_prompt_logprobs=True to get per-token log-probabilities for the input tokens alongside the output, and set topk_prompt_logprobs above 0 to also return the top-k alternatives at each prompt position. The client selects the base model and adapter, so there is no per-call model override.The first request against a sampler deployment waits for its deployment to be ACTIVE before sending, with the same semantics as ensure_ready: a scaled-to-zero sampler gets one wake, and a deployment in a terminal-failure state raises SamplerTerminatedError. If your API key can’t read deployment status, the check is skipped and a failed sampler surfaces as request errors instead. Serverless samplers have no per-run deployment to poll or wake.Set timeout on ServiceClient.create_sampling_client, or sampler_timeout on create_lora_training_client, to bound a sampling response in the dedicated mode or the wait for a serverless sampling result. The default is no timeout. When the timeout expires, sample() raises SampleTimeoutError without retrying the generation. For serverless samples, the SDK also attempts to cancel work that’s still queued. A timeout doesn’t deactivate the run or sampler.For sampler deployments, sample() retries transient 503 responses and 408 responses from replicas below the requested policy version, bounded by ready_timeout. It doesn’t retry a 504.For serverless samplers, the SDK submits a queued operation and polls its result. Submission retries use the same idempotency key. It retries transient 408, 502, and 504 responses; it retries 503 only when the response includes Retry-After. Gateway rejections and invalid requests fail without readiness retries. See Handle SDK request errors.affinity_key (keyword-only) is sent as the X-Baseten-Session-ID header so sticky-routing-enabled Baseten deployments send related requests to the same sampler replica, keeping a rollout group’s shared prompt in one replica’s prefix cache. Reuse one key across every rollout and turn in a rollout group; don’t reuse a single key for an entire run. Must be a non-empty string when set.include_routed_experts controls expert-route capture on router replay models. Pass True to capture the routes with each recorded sample, which requires a client created with record_stability_info=True, or False to opt out. The default, None, preserves the sampler’s configured behavior.list[float | None]
Return the per-token log-probabilities for
prompt without generating any new tokens. Index 0 is always None because the first token has no preceding context to score against. Other positions may also be None if the sampler can’t compute a log-probability for that token. affinity_key works exactly as in sample(); scoring a prompt is a full prefill, so passing the rollout group’s key routes the request to the replica whose prefix cache already holds the group’s shared prompt.str
Return the base model ID. For serverless samplers, returns the model this client was created for. For sampler deployments, reads the
/v1/models entry with no parent and retries with backoff while the sampler is still deploying.str | None
For serverless samplers, return the checkpoint path this client samples from, or
None for a base-model client. For sampler deployments, return the first /v1/models entry with a non-null parent, or None if no adapter is loaded.str
Return the base model ID this sampling client was created with, without contacting the server.
str | None
Property. The
bt:// checkpoint path this client is pinned to, or None when it serves the base model. For a serverless sampler, this is the unique adapter name the publishing call minted ({name}-<uuid>), not the name that call was given.str | None
Property. The name
checkpoint_path was published under, or None. Pass it to get_checkpoint_archive_url(name=...) or download_checkpoint(name=...) for the save that created this client. On serverless capacity it can differ from the name you passed to save_weights_and_get_sampling_client.PreTrainedTokenizer
Return the Hugging Face
PreTrainedTokenizer for the base model this client was created with, or for the pinned custom base checkpoint when the client serves one.None
Block until a sampler deployment’s status is
ACTIVE. A scaled-to-zero sampler triggers one wake; terminal-failure states raise SamplerTerminatedError. No-op for local and serverless samplers, which have no deployment status to poll.None
Class method. Block until
deployment reports ready, using a throwaway SamplingClient so you can wait without holding one. Polls up to ready_timeout seconds and applies the same readiness semantics as ensure_ready.None
Close the client’s HTTP connections and local event-loop resources. This doesn’t deactivate the sampler. To end a paired sampler, deactivate its run. To end a standalone sampler, deactivate the sampler.
None
Async counterpart to
close() for callers running their own event loop.