Model performance overview

Tradeoff: Latency/Throughput/Cost vs Quality

The biggest factor in your latency, throughput, cost, and quality is what model you use. Before you jump into optimizing a foundation model, consider:

• Can you use a smaller size, like Llama 8B instead of 70B? Can you fine-tune the smaller model for your use case?
• Can you use a different model, like SDXL Lightning instead of SDXL?
• Can you use a different implementation, like Faster Whisper instead of Whisper?

Usually, model selection is bound by quality. For example SDXL Lightning makes images incredibly quickly, but they may not be detailed enough for your use case.

Experiment with alternative models to see if they can reset your performance expectations while meeting your quality bar.

Tradeoff: Latency/Throughput vs Cost

The minimum requirement for a GPU instance is that it must have enough VRAM to load model weights with headroom left for inference.

It often makes sense to use a more powerful (but more expensive) GPU than the minimum requirement, especially if you have ambitious latency goals and/or high utilization.

For example, you might choose:

• (Multiple) H100 GPUs for deployments optimized with TensorRT/TensorRT-LLM
• H100 MIGs for high-throughput deployments of smaller models like Llama 3 8B and SDXL
• L4 GPUs for autoscaling Whisper deployments

The GPU instance reference lists all available options.

Benefit: Latency/Throughput/Cost

You can just use transformers and pytorch out of the box to serve your model. But best-in-class performance comes from using a dedicated inference engine, like:

1. TensorRT/TensorRT-LLM, maintained by NVIDIA
2. vLLM, an independent open source project
3. TGI, maintained by Hugging Face

We often recommend TensorRT/TensorRT-LLM for best performance. The easiest way to get started with TensorRT-LLM is our TRT-LLM engine builder.

Benefit: Latency/Throughput

In addition to an optimized inference engine, you need an inference server to handle requests and supply features like in-flight batching.

Baseten runs a modified version of Triton for compatible model deployments. Other models use TrussServer, a capable general-purpose model inference server built into Truss.

Tradeoff: Latency/Throughput/Cost vs Quality

By default, model inference happens in fp16, meaning that model weights and other values are represented as 16-bit floating-point numbers.

Through a process called post-training quantization, you can instead run inference in a different format, like fp8, int8, or int4. This has massive benefits: more teraFLOPS at lower precision means lower latency, smaller numbers being retrieved from VRAM means higher throughput, and smaller model weights means saving on cost and potentially using fewer GPUs.

However, quantization can affect output quality. Thoroughly review quantized model outputs by hand and with standard checks like perplexity to ensure that the output of the quantized model matches the original.

We’ve had a lot of success with fp8 for faster inference without quality loss and encourage experimenting with quantization, especially when using the TRT-LLM engine builder.

Tradeoff: Latency/Throughput/Cost vs Quality

There are a number of exciting cutting-edge techniques for model inference that can massively improve latency and/or throughput for a model. For example, LLMs can use Speculative Decoding or Medusa to generate multiple tokens per forward pass, improving TPS.

When using a new technique to improve model performance, always run real-world benchmarks and carefully validate output quality to ensure the performance improvements aren’t undermining the model’s usefulness.

Tradeoff: Latency vs Throughput/Cost

Batch size is how many requests are processed concurrently on the GPU. It is a direct tradeoff between latency and throughput:

• Increase batch size to improve throughput and cost
• Reduce batch size to improve latency

Tradeoff: Latency/Throughput vs Cost

If traffic to a deployment is high enough, even an optimized model server won’t be able to keep up. By creating replicas, you keep latency consistent for all users.

Learn more about autoscaling model replicas.

Benefit: Latency

A “cold start” is the time it takes to spin up a new instance of a model server. Fast cold starts are essential for useful autoscaling, especially scale to zero.

Read our guide to improving cold start times for options like caching model weights.

Tradeoff: Latency vs Throughput/Cost

Replica-level concurrency sets the number of requests that can be sent to the model server at one time. This is different from the on-GPU concurrency as your model server may perform pre- and post-processing tasks on CPU.

Replica-level concurrency should always be greater than or equal to on-device concurrency (batch size).

Tradeoff: Latency vs Cost

If your GPU is in us-east-1 and your customer is in Australia, it doesn’t matter how much you’ve optimized TTFT — your real-world latency will be terrible.

Region-specific deployments are available on a per-customer basis. Contact us at support@baseten.co to discuss your needs.

Benefits: Latency, Quality

Every token an LLM doesn’t have to process or generate is a token that you don’t have to wait for or pay for.

Prompt engineering can be as simple as saying “be concise” or as complex as making sure your RAG system returns the minimum number of highly-relevant retrievals.

Benefits: Latency, Throughput

When using TensorRT-LLM, make sure that your input and output sequences are a consistent length. The inference engine is built for a specific number of tokens, and going outside of those sequence shapes will hurt performance.

Benefits: Latency, Cost

The only thing running on your GPU should be the AI model. Other tasks like retrievals, secondary models, and business logic should be deployed and scaled separately to avoid bottlenecks.

Use Chains for performant multi-step and multi-model inference.

Benefit: Latency

Use sessions rather than individual requests to avoid unnecessary network latency. See inference documentation for details.