Skip to main content
Different traffic patterns require different autoscaling configurations. Identify your pattern below for recommended starting settings.
These are starting points, not final answers. Monitor your deployment’s performance and adjust based on observed behavior. See Autoscaling for parameter details.

​
Identify your pattern

Not sure which pattern you have? Check your metrics:
  1. Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar, then select your model.
  2. Choose the Metrics tab.
  3. Look at Inference volume and Replicas over the past week.
  4. Compare to the patterns below.
If traffic combines a daily pattern with sharp bursts, tune for the bursts, then measure cost during steady periods.

​
Jittery traffic

Small, frequent spikes that quickly return to baseline.

​
Characteristics

  • Traffic repeatedly rises to at least twice its baseline, then quickly returns.
  • Spikes are short-lived and return to baseline quickly.
  • Often not real load growth, just temporary surges causing overreaction.

​
Common causes

  • Consumer products with intermittent usage bursts.
  • Traffic splitting or A/B testing with low percentages.
  • Polling clients with synchronized intervals.
A longer autoscaling window averages out the jitter so the autoscaler doesn’t react to every small spike. This trades reaction speed for stability when the spikes aren’t sustained load increases.
If you’re still seeing oscillation with these settings, increase the scale-down delay before lowering target utilization.

​
Bursty traffic

​
Characteristics

  • Traffic jumps sharply (2x+ within 60 seconds).
  • Stays high for a sustained period before dropping.
  • Queuing and latency increase while replicas start.

​
Common causes

  • Daily morning ramp-up (users starting their day).
  • Marketing events, product launches, viral moments.
  • Top-of-hour scheduled jobs or cron-triggered traffic.
A shorter window lets the autoscaler react sooner. A longer delay prevents scale-down between waves. Lower target utilization leaves capacity available while new replicas start.

​
Pre-warming for predictable bursts

To pre-warm before a predictable burst (morning ramp, scheduled events):
  1. Before the expected spike, bump min replicas:
    Request
  2. After the burst subsides, reset to your normal minimum:
    Request
For peaks at known times, use an autoscaling schedule to raise minimum replicas before traffic arrives. Start the schedule early enough to cover your deployment’s observed cold start time.

​
Scheduled traffic

​
Characteristics

  • Long periods of low or zero traffic.
  • Large bursts tied to job schedules (hourly, daily, weekly).
  • Traffic patterns are predictable but infrequent.

​
Common causes

  • ETL pipelines and data processing jobs.
  • Embedding backfills and batch inference.
  • Periodic evaluation or testing jobs.
  • Document processing triggered by user uploads.
Scale-to-zero avoids replica charges during idle periods. The moderate window filters short-lived changes at the start of a batch. If jobs come in waves, a longer delay keeps replicas warm between them.

​
Scheduled pre-warming

Autoscaling schedules pre-warm replicas without an external job or API key. For a batch job that starts at the top of each hour, create an hourly window that starts at :55 and ends after the job completes. Set the schedule’s min_replica to the warm capacity the job needs. Outside the window, the environment returns to its default settings. Start the window earlier if replicas take more than five minutes to become ready. Use your worst observed cold start as the initial lead time, then adjust it from deployment metrics.
If an existing orchestration system must own the schedule, run the same updates through the Management API. This example raises the minimum five minutes before an hourly job:
Terminal
This update restores scale-to-zero 30 minutes after the hour:
Terminal
If the deployment has reached zero replicas when a batch starts, Baseten must cold-start a replica before it can process the batch. Synchronous requests can wait while the replica starts. Setting min_replica to 0 allows the deployment to reach this state. For latency-sensitive batch jobs, set min_replica to at least 1 for the job window and start pre-warming before traffic arrives.

​
Steady traffic

​
Characteristics

  • Traffic rises and falls gradually over the day.
  • Classic diurnal pattern with no sharp edges.
  • Predictable, cyclical behavior.

​
Common causes

  • Always-on inference APIs with consistent user base.
  • B2B applications with business-hours usage.
  • Production workloads with stable, mature traffic.
Without sudden spikes, you don’t need as much headroom. You can run replicas at higher utilization to reduce cost because load changes are gradual and predictable. The autoscaler has time to react.
Start with the defaults. Monitor for a week, then gradually raise target utilization while watching p95 latency.

​
Next steps