Skip to main content
A finished run produces checkpoints, not a better model. Shipping one means showing it beats what you serve today on data it never trained on, deploying it with the settings you evaluated, and keeping a path back.

​
Confirm the run learned

Check the training metrics before you spend time evaluating a checkpoint:
  • SFT: validation loss should fall with training loss. Validation loss that flattens is normal near the end of a run. Validation loss that rises while training loss keeps falling means the model is memorizing the training examples; use an earlier checkpoint or train for fewer steps.
  • RL: reward should rise on groups the run trained, without a matching rise in rejected groups or failed attempts, and transcripts should show the behavior you wanted. See Check whether an RL run is learning.
Save checkpoints along the way, not only at the end. The best checkpoint on held-out data isn’t always the last one.

​
Compare with the baseline and the incumbent

Score the checkpoint on your frozen held-out set with the same evaluation you used to choose the base model. Compare it with two models:
  • The untrained base model shows what training added.
  • The model you serve today shows whether the checkpoint is ready to replace it.
Read every check separately, and read the worst regressions row by row. A gain on the task can come with a loss somewhere else, such as answering correctly in the wrong format or no longer refusing requests it should refuse. Keep a slice of held-out cases for behavior the current model already gets right, so a regression shows up as a number. Decide against the thresholds you set before training, not ones chosen after seeing the scores. See Evaluate a model.

​
Deploy with the settings you evaluated

Deploy the checkpoint as an inference deployment that serves the base model with your adapter. Send it the same system prompt, tool definitions, and chat template settings, including the reasoning setting, that the training data and evaluation used. A model trained and evaluated with one system prompt and served with another is running a task it wasn’t tested on. Score the deployment itself on the held-out set before it takes traffic. That catches serving differences, such as a tool-call parser or template setting, that a sampler checkpoint wouldn’t show.

​
Roll it out with a way back

Move traffic in steps and keep the current model ready to take it back:
  • The new deployment belongs to the same Baseten model as the current one: promote it to the environment with a rolling deployment, which shifts traffic incrementally and can be paused or cancelled. Set the environment’s cleanup strategy to KEEP or SCALE_TO_ZERO so the previous deployment stays available to promote back.
  • The new deployment is a separate model: switch traffic in your application. Send a share of requests to the new model first and compare outcomes on live traffic before you move the rest.
Delete deployments you no longer need. They bill for GPU time while they’re live, and checkpoints stay available for a later redeploy.

​
Feed production back into the next run

A deployed model stops improving unless you feed it new data. Keep the loop going:
  • Log inputs and outputs for the task, with any corrections people make to them. Corrections are the strongest training targets you can collect.
  • Add every production failure to the evaluation. A failure that becomes a held-out case can’t regress silently in the next model.
  • Retrain when the evaluation shows a new gap. Build the next dataset or environment from the logged traffic, train from the current base model or checkpoint, and compare the result with the model you now serve.

​
Next steps