​
Overview

A/B deployments are a release process where a new deployment (the “candidate” or “B”) is gradually released to a subset of users alongside the existing deployment (the “production” or “A”). This controlled rollout helps validate the quality and performance of the new deployment before wider release.

​
Benefits

  • Controlled Validation: Direct a small percentage of users to experience a new model, collecting both product and performance metrics before full rollout
  • Traffic Management: Mitigate risks associated with traffic spikes to models that haven’t warmed up to full capacity
  • Resource Optimization: Control the pace and capacity allocated to new models, helping manage resources during large migrations that would otherwise require nearly double the existing capacity
  • Safety Net: Maintain an active fallback deployment in case issues arise with the new model

A/B rollouts provide a more manual but controlled approach compared to canary deployments.

​
Implementation Requirements

To implement A/B deployments with Baseten, you’ll need to create routing logic on your side that directs traffic to either deployment A (production) or B (candidate) based on a configurable threshold. Below are two approaches to implementing this routing logic.

​
Approach 1: Random Request Routing

This approach randomly routes each individual request to either deployment A or B based on a probability threshold.

import random
import os

def get_model_url():
    """
    Returns either the production or candidate URL based on a random probability.

    Returns:
        URL string for the selected deployment
    """
    # Configuration
    PRODUCTION_URL = os.environ.get("PRODUCTION_URL") # e.g."<https://my-model-id.api.baseten.co/environments/production/predict>"
    CANDIDATE_URL = os.environ.get("CANDIDATE_URL") # e.g. "<https://my-model-id.api.baseten.co/deployment/my-deployment-id/predict>"
    CANDIDATE_PERCENTAGE = float(os.environ.get("CANDIDATE_PERCENTAGE", "0.05"))  # 5% to candidate by default

    # Determine which deployment to use
    random_value = random.random()  # Generate a random value between 0 and 1
    use_candidate = random_value < CANDIDATE_PERCENTAGE

    # Return the appropriate URL
    selected_url = CANDIDATE_URL if use_candidate else PRODUCTION_URL

    # Log which deployment is being used (optional)
    deployment_type = "CANDIDATE" if use_candidate else "PRODUCTION"
    print(f"Selected {deployment_type} deployment")

    return selected_url

# Example usage:
# url = get_model_url()
# Then use this URL with your preferred HTTP client

​
Approach 2: Session-Based Routing

For use cases requiring consistency across multiple requests, session-based routing ensures all related requests go to the same deployment. This approach is particularly valuable for:

  • Contextual Continuity: Essential for multi-turn conversations or chat sessions where context is built up over time
  • User Experience Consistency: Prevents jarring changes in model behavior or quality mid-session
  • Quality Evaluation: Simplifies A/B testing by ensuring entire user journeys can be attributed to a specific model version
  • Context Window Efficiency: For models that maintain conversation history, keeping users on the same deployment prevents redundant context transmission and increases the chance of a cache hit
import hashlib
import os

def get_model_url(session_id):
    """
    Returns either the production or candidate URL based on a stable session ID hash.

    Args:
        session_id: A unique identifier for the user session (conversation ID, user ID, etc.)

    Returns:
        URL string for the selected deployment
    """
    # Configuration
    PRODUCTION_URL = os.environ.get("PRODUCTION_URL") # e.g."<https://my-model-id.api.baseten.co/environments/production/predict>"
    CANDIDATE_URL = os.environ.get("CANDIDATE_URL") # e.g. "<https://my-model-id.api.baseten.co/deployment/my-deployment-id/predict>"
    CANDIDATE_PERCENTAGE = float(os.environ.get("CANDIDATE_PERCENTAGE", "0.05"))  # 5% to candidate by default

    # Create a deterministic hash of the session ID
    hash_obj = hashlib.md5(session_id.encode())
    # Convert hash to a float between 0 and 1
    hash_float = int(hash_obj.hexdigest(), 16) / (2**128 - 1)

    # Determine which deployment to use
    use_candidate = hash_float < CANDIDATE_PERCENTAGE

    # Return the appropriate URL
    return CANDIDATE_URL if use_candidate else PRODUCTION_URL

# Example usage:
# url = get_model_url("user-123456")
# Then use this URL with your preferred HTTP client

​
Session-Based Routing Considerations

  • Distribution Variance: The actual traffic distribution may not exactly match your target percentage, especially when some sessions contain more “events” than other sessions.
  • Consistency: Session IDs should consistently route to the same deployment even as you adjust the threshold, ensuring a session changes deployments at most once during the migration.
  • Tracking: Record which deployment handled each request or session to facilitate analysis and troubleshooting.
  • Entropy of Session ID: ensure the ID used for your session has sufficiently high entropy to ensure uniform distribution of requests across models. Standard UUIDs will work.

​
Baseten Release Process with A/B Deployments

Once you’ve added A/B support to your code, follow these steps to roll out a new deployment.

Prerequisite: You should already have an existing model with a production deployment on Baseten.

  1. Deploy the new model version: Ensure it doesn’t have environment-specific dependencies that would cause behavioral differences from your production model (unless intended).
  2. Configure your routing URLs:
    • Deployment A (Production): Use the production environment alias
    • https://{model-id}.api.baseten.co/environments/production/predict
    • Deployment B (Candidate): Use the specific deployment ID
    • https://{model-id}.api.baseten.co/deployment/{deployment-id}/predict
  3. Scale appropriately: Pre-scale deployment B to handle its expected traffic share. For example, if testing with 10% of traffic, set the minimum replica count to at least 10% of deployment A’s capacity.
  4. Activate A/B routing: Enable your routing logic and begin directing traffic to both deployments.
  5. Monitor performance: Track both quantitative metrics (through the Baseten Metrics dashboard) and qualitative results from your product (success rates, error rates, etc.).
  6. Manage capacity: As traffic shifts to deployment B, deployment A should auto-scale down if properly configured. This will free up capacity.
  7. Complete the migration: Repeat the process as many times as you’d like, gradually increasing the % of traffic routed to your candidate deployment. When ready to fully switch, simply promote deployment B to production within Baseten. At this point:

Both URLs will point to the same production environment (one via the production alias, and the other via the deployment ID)

Update the minimum/maximum replica count for your new production deployment to match your previous production environment values (or just let the system do this automatically during promotion)

Configure your routing to direct 100% of traffic to the production deployment using the production alias

By following this guide, you can safely and effectively roll out new model deployments while minimizing risk and optimizing resource usage.