Traffic patterns

Different traffic patterns require different autoscaling configurations. Identify your pattern below for recommended starting settings.

​

Identify your pattern

Not sure which pattern you have? Check your metrics:

1. Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar, then select your model.
2. Choose the Metrics tab.
3. Look at Inference volume and Replicas over the past week.
4. Compare to the patterns below.

​

Jittery traffic

Small, frequent spikes that quickly return to baseline.

​

Characteristics

• Baseline replica count is steady, but spikes up by 2x several times per hour.
• Spikes are short-lived and return to baseline quickly.
• Often not real load growth, just temporary surges causing overreaction.

​

Common causes

• Consumer products with intermittent usage bursts.
• Traffic splitting or A/B testing with low percentages.
• Polling clients with synchronized intervals.

​

Recommended settings

A longer autoscaling window averages out the jitter so the autoscaler doesn’t chase every small spike. You’re trading reaction speed for stability, which is acceptable when the spikes aren’t sustained load increases.

​

Bursty traffic

​

Characteristics

• Traffic jumps sharply (2x+ within 60 seconds).
• Stays high for a sustained period before dropping.
• The “pain” is queueing and latency spikes while new replicas start.

​

Common causes

• Daily morning ramp-up (users starting their day).
• Marketing events, product launches, viral moments.
• Top-of-hour scheduled jobs or cron-triggered traffic.

​

Recommended settings

Short window means fast reaction. Long delay prevents scaling down between waves. Lower utilization gives you buffer capacity while new replicas start.

​

Pre-warming for predictable bursts

To pre-warm before a predictable burst (morning ramp, scheduled events):

1. Before the expected spike, bump min replicas:
   Request

   curl -X PATCH \
     https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \
     -H "Authorization: Bearer $BASETEN_API_KEY" \
     -d '{"min_replica": 5}'
   
2. After the burst subsides, reset to your normal minimum:
   Request

   curl -X PATCH \
     https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \
     -H "Authorization: Bearer $BASETEN_API_KEY" \
     -d '{"min_replica": 2}'
   

​

Scheduled traffic

​

Characteristics

• Long periods of low or zero traffic.
• Large bursts tied to job schedules (hourly, daily, weekly).
• Traffic patterns are predictable but infrequent.

​

Common causes

• ETL pipelines and data processing jobs.
• Embedding backfills and batch inference.
• Periodic evaluation or testing jobs.
• Document processing triggered by user uploads.

​

Recommended settings

Scale-to-zero saves significant cost during idle periods. The moderate window prevents overreacting to the initial requests of a batch. If jobs come in waves, a longer delay keeps replicas warm between them.

​

Scheduled pre-warming

To pre-warm for predictable batch jobs, use cron + API:

1. Five minutes before the hourly job, scale up:
   Terminal

   0 * * * * curl -X PATCH \
     https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \
     -H "Authorization: Bearer $BASETEN_API_KEY" \
     -d '{"min_replica": 3}'
   
2. Thirty minutes after the job completes, scale back down:
   Terminal

   30 * * * * curl -X PATCH \
     https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \
     -H "Authorization: Bearer $BASETEN_API_KEY" \
     -d '{"min_replica": 0}'
   

​

Steady traffic

​

Characteristics

• Traffic rises and falls gradually over the day.
• Classic diurnal pattern with no sharp edges.
• Predictable, cyclical behavior.

​

Common causes

• Always-on inference APIs with consistent user base.
• B2B applications with business-hours usage.
• Production workloads with stable, mature traffic.

​

Recommended settings

Without sudden spikes, you don’t need as much headroom. You can run replicas at higher utilization (lower cost) because load changes are gradual and predictable. The autoscaler has time to react.

​

Next steps

• Autoscaling: Full parameter documentation.
• Scale a deployment: Apply these settings from the CLI or API.
• Troubleshooting autoscaling: Diagnose and fix common problems.
• Truss configuration reference: Configure predict_concurrency in your model.