Together AI 推出金丝雀发布，支持生产环境无停机升级模型

Summary

Rollouts move live traffic from your current model to a new checkpoint in gated steps. Health checks always run before traffic moves; on a canary you can also add metric gates (say p95 latency or error rate) that run after each step. If a gate trips, the rollout pauses at the canary share and you cancel it and run it in reverse. Below we run one for real: a Qwen2.5-7B → Qwen3.5-9B canary whose gate caught a 137% p95 regression at 10% of traffic; we canceled and reversed it with live requests served and none failed.

If you run a model in production, you already know the need to swap in a new checkpoint or a new model family: the open model ecosystem moves fast, and the candidate usually looks great in evals or promises better throughput. You want it in front of every user without hiccups, and a way to rollback if it disappoints. The usual options force a tradeoff:

Both of these options put a human in the loop as the safety mechanism. Rollouts move that mechanism into the platform: you describe the source, the target, the steps, and what "healthy" means, and the platform works against this plan at every stage.

A rollout migrates traffic between two deployments on the same endpoint: a source (what's serving today) and a target (what you want to serve tomorrow). You pick one of three strategies:

Here's what happens inside every canary step:

We chose this ordering deliberately; each item prevents a class of incidents:

Through the API or the console, a rollout is created in a PENDING state and does nothing until you explicitly start it (the CLI's rollout command creates and starts in one step). This two-step create/start is intentional because you can create the rollout, review it (or have a teammate review it), and start it when you're actually watching.

Two states in the diagram above deserve a note:

There is no FAILED end state that leaves traffic in limbo: a rollout ends COMPLETED (the target serves) or CANCELED (the split is frozen where it was, and you run the rollout in reverse to go back).

The following is a breakdown of what happens in a single canary step, measured on the run at the end of this post (Qwen2.5-7B → Qwen3.5-9B on one H100 each).

The propagation wait is what keeps stale global routing caches from sending requests to a shrinking source. The wait period is grown to the metric window plus ingestion lag. The cold start dominates the first step; later steps add replicas to a target that is already serving and warm.

All three strategies run through the same engine and the same health gates; they differ in how traffic moves, how much extra capacity the overlap costs, and whether there is a wait window for a metric gate.

‍

Here's a three-step canary from a deployment serving your current model to one serving the candidate, with a latency regression gate. The CLI ships as tg in the together Python package (2.34.0 or newer). You pass the target deployment; the source is inferred when exactly one deployment is receiving traffic, otherwise pass --source :

# 1. Create AND start the rollout in one command. # Intervals and windows are seconds with an "s" suffix ("600s", not "10m"). tg beta endpoints rollout $TARGET_DEPLOYMENT_ID \ --source $SOURCE_DEPLOYMENT_ID \ --canary \ --steps 10,50,100 \ --interval 600s \ --metric router_latency --metric-stat p95 \ --metric-max-regression 10 --metric-direction higher-is-worse \ --metric-window 300s

# 2. Watch it move: pass the rollout ID printed under "Active Rollout", # or the endpoint ID for the endpoint summary tg beta endpoints get $ROLLOUT_ID tg beta endpoints get $ENDPOINT_ID

# 3. Control it: pass the endpoint ID plus exactly one control flag tg beta endpoints rollout $ENDPOINT_ID --pause --reason "holding for review" tg beta endpoints rollout $ENDPOINT_ID --resume tg beta endpoints rollout $ENDPOINT_ID --promote tg beta endpoints rollout $ENDPOINT_ID --cancel --reason "latency regression on target"

The source drains to zero replicas and stops when the rollout completes ( --final-source-replicas defaults to 0), and the target lands with the source's replica count as its floor ( --final-target-replicas ). The CLI attaches one metric gate per rollout; for several rules use the console or the API.

The same via the REST API, where create and start are separate calls and a rollout can carry several metric rules:

# 1. Create the rollout. It comes back in state PENDING; save its "id" (rol_...) as $ROLLOUT_ID. curl -s -X POST \ "https://api.together.ai/v2/projects/$PROJECT_ID/endpoints/$ENDPOINT_ID/rollouts" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "sourceDeploymentId": "'$SOURCE_DEPLOYMENT_ID'", "targetDeploymentId": "'$TARGET_DEPLOYMENT_ID'", "canary": { "steps": [{"traffic": 10}, {"traffic": 50}, {"traffic": 100}], "stepInterval": "600s" }, "metrics": [{ "name": "router_latency", "stat": "METRIC_STAT_TYPE_PERCENTILE", "percentile": 95, "regressionCheck": { "direction": "REGRESSION_DIRECTION_HIGHER_IS_WORSE", "maxRegressionPercent": 10 }, "window": "300s" }] }'

# 2. Start it: POST .../rollouts/$ROLLOUT_ID/start -d '{}' # 3. Watch it: GET .../rollouts/$ROLLOUT_ID

A few things the API is strict about: percentile is an integer (95, not "p95"), enum values carry their full prefix ( METRIC_STAT_TYPE_* , REGRESSION_DIRECTION_* , THRESHOLD_OPERATOR_* ), durations are protobuf strings like "600s" , and a metric name outside the catalog is rejected with a 400 that lists the supported names. Draining the source is the default, so there is nothing to pass for it.

The regression check can be understood as: at each gate, compare the target's p95 router latency (the per-request duration measured at the router, in milliseconds) over the last 5 minutes against the source's. If the target is more than 10% worse, don't proceed.

The same rollout from Python, with the together package (2.34.0 or newer). Field names are snake_case here and camelCase on the wire; the SDK translates.

from together import Together

client = Together() # reads TOGETHER_API_KEY

rollout = client.beta.endpoints.rollouts.create( endpoint_id=ENDPOINT_ID, project_id=PROJECT_ID, source_deployment_id=SOURCE_DEPLOYMENT_ID, target_deployment_id=TARGET_DEPLOYMENT_ID, canary={"steps": [{"traffic": 10}, {"traffic": 50}, {"traffic": 100}], "step_interval": "600s"}, metrics=[{ "name": "router_latency", "stat": "METRIC_STAT_TYPE_PERCENTILE", "percentile": 95, "regression_check": {"direction": "REGRESSION_DIRECTION_HIGHER_IS_WORSE", "max_regression_percent": 10}, "window": "300s", }], ) # state: PENDING

client.beta.endpoints.rollouts.start(rollout.id, project_id=PROJECT_ID, endpoint_id=ENDPOINT_ID)

# later: .retrieve / .pause / .resume / .promote / .cancel take the same (id, project_id, endpoint_id)

Controlling a rollout Every rollout accepts the same four controls. An endpoint has at most one active rollout, so the CLI takes the endpoint ID and you rarely need the rollout ID. Each control returns as soon as it is accepted; poll tg beta endpoints get (or the GET endpoint) until the rollout reaches the state you expect. While a rollout is active, including while paused, the endpoint's traffic split is locked and its source and target cannot be stopped or deleted.

The rollout goes PAUSING , lets any step activity in flight finish, then holds at the current traffic split and replica counts as PAUSED . Both deployments keep serving. A pause can last for days; the platform never auto-resumes an operator pause.

tg beta endpoints rollout $ENDPOINT_ID --pause --reason "holding for review" REST POST …/rollouts/$ROLLOUT_ID/pause {"reason": "holding for review"}

Resume Continues from the same step, for both PAUSED and SYSTEM_PAUSED . If a gate tripped, it re-evaluates against fresh data; the step is not skipped.

tg beta endpoints rollout $ENDPOINT_ID --resume REST POST …/rollouts/$ROLLOUT_ID/resume {}

Promote Skips the remaining canary steps and runs the final 100% step in full: the target scales to its landing size, traffic shifts, the propagation wait and soak run, then the source drains. Skipped steps are recorded as SKIPPED . Not instantaneous: in a test run with a 10-minute step interval, a promote at step 0 still sat through the final step's full soak.

tg beta endpoints rollout $ENDPOINT_ID --promote REST POST …/rollouts/$ROLLOUT_ID/promote {}

Cancel Freezes the current traffic split into the endpoint's standing weights and ends the rollout as CANCELED . Nothing scales down; both deployments keep serving their frozen shares until you edit the split or run a reverse rollout. A target canceled at 0% is left running with no traffic; scale it to zero or delete it if you no longer need it.

tg beta endpoints rollout $ENDPOINT_ID --cancel --reason "latency regression on target" REST POST …/rollouts/$ROLLOUT_ID/cancel {"reason": "latency regression on target"}

Reverse rollout There is no rollback verb. To move traffic back, after a cancel or after a completion, create a new rollout with source and target swapped, then start it. Any strategy works and the same gates apply. After a cancel the default canary ladder skips the steps the new target has already passed, and the default final replica count is the pair's combined count.

tg beta endpoints rollout $OLD_SOURCE_ID --source $OLD_TARGET_ID --canary REST POST …/rollouts (with the two IDs swapped) POST …/rollouts/$NEW_ROLLOUT_ID/start

Controls on a finished rollout ( COMPLETED or CANCELED ) are refused. Delete a finished or never-started rollout from the history with tg beta endpoints rm $ROLLOUT_ID ; deleting the record does not change the traffic split it left behind.

Metric gates are a canary feature: blue-green and rolling still run health gates, but the staged metric comparison needs canary's step structure to be meaningful. Gates evaluate over a closed catalog of three router-side metrics, measured identically for source and target (any other metric name is rejected at create time):

Each rule uses one of two checks:

Two recipes cover most services:

// Latency-sensitive service: don't ship anything meaningfully slower { "name": "router_latency", "stat": "METRIC_STAT_TYPE_PERCENTILE", "percentile": 95, "regressionCheck": { "direction": "REGRESSION_DIRECTION_HIGHER_IS_WORSE", "maxRegressionPercent": 10 }, "window": "300s" }, // Error-budget shop: hard ceiling regardless of what the source is doing { "name": "router_error_rate", "stat": "METRIC_STAT_TYPE_AVG", "thresholdCheck": { "operator": "THRESHOLD_OPERATOR_LT", "value": 0.01 }, "window": "300s" }

Timing windows Three durations interact, so keep all of them in mind:

You must wait for a period of at least window + ingestion lag , so that the gate's entire lookback period lands inside the current step's steady state. If you wait for a shorter period than your window, the gate would be comparing metrics that partially describe the previous traffic split. The platform enforces this for you: if you request a wait period that's too short for your window, it increments it automatically. It is still better to design with it in mind: a 5m window needs a 6.5m wait period. With the default 5m window the platform grows the default 3m interval to 390s (6m 30s); if you set your own stepInterval , make it at least window + 90s .

By default, a tripped gate routes to SYSTEM_PAUSED , which means the system pauses for review . The rollout holds at its current split (the blast radius stays at whatever your canary percentage was), and you decide: resume (the gate re-evaluates), promote, or cancel.

There is no automatic abort: a confirmed regression always parks the rollout for a human, because moving traffic back is itself a change someone should be watching. Recoverable causes such as a capacity shortfall or a metrics-pipeline gap are different: the platform retries those every 15 minutes for up to 3 hours before leaving the rollout paused for you. The platform also guards against false alarms: before it pauses on a regression the system re-queries several times over ~90 seconds to make sure it isn’t looking at ingestion lag or transient blips, and a gate that cannot get trustworthy data pauses with METRICS_UNAVAILABLE rather than counting as a regression.

All three strategies run through the same step engine, so these hold for canary, blue-green and rolling alike.

1. Capacity is never rounded down. Target replicas round up and the source drain rounds down, so a same-size swap never has fewer replicas than it started with. Rolling adds one replica mid-step; blue-green briefly runs both deployments at full size. Replicas your autoscaler added above the plan are kept.

2. Traffic never lands on capacity that is not ready. Every step runs in one order: scale the target, check health, shift traffic, wait 30 s for routing to converge, drain the source, wait, evaluate the gate, record the step. A step that regressed during its wait is never recorded as passed.

3. Each side always has the replicas its share needs. Traffic moves only once the target has enough ready replicas for the new share, and the source is never drained below its remaining share. If a replica dies and the split can no longer be served, the rollout holds the largest split it can and pauses as UNDER_SERVED .

4. The rollout raises floors; it does not fight your autoscaler. Each step writes each deployment's minimum replicas and nothing else, with one exception: the target's maximum is lifted once so it can carry the whole endpoint, and stays lifted. The source's maximum shrinks with its share during the drain. Lower a maximum below what the step needs and the rollout pauses as POLICY_INFEASIBLE instead of overriding you.

5. The gate always returns a verdict. A regression check passes when the target is within your percentage budget of the source. No source data passes; a zero source against a non-zero target on a higher-is-worse metric fails; any other zero source passes. A threshold check ignores the source and compares the target with your value.

6. Gates read only the current step's traffic, and only enough of it. The wait period is at least the metric window plus about 90 s of ingestion lag, so a 300 s window means a 390 s wait. p95 needs 20 requests in the window and p99 needs 100; with fewer the rollout pauses as METRICS_UNAVAILABLE . Error rate and in-flight requests need one.

1. What if there's no GPU capacity for the target?

The rollout checks feasibility for the entire journey up front, before touching anything, and again at each scale-up. A shortfall pauses the rollout in SYSTEM_PAUSED with a CAPACITY_EXHAUSTED category. At this point nothing has moved and your source is untouched. Resume re-checks capacity and continues if it's freed up. Capacity problems are usually transient, so pausing beats failing.

2. Can I pause indefinitely?

Yes. Pause is not a held connection but rather a first-class state. Rollouts are designed to survive multi-day pauses and resume exactly where they left off.

When the platform pauses a rollout, status.condition carries a typed failure category plus a human-readable message . These include:

The docs list the remaining categories and what to do about each: Troubleshooting rollouts .

Everything above is easier to trust after watching it in action once, so here is a run on the current platform (September 2026). We upgraded a live endpoint from Qwen2.5-7B-Instruct to Qwen3.5-9B , each on a single H100, while a steady 5 requests per second of chat completions flowed through the endpoint the whole time and every response code was logged. The newer model is the one we wanted; the question a rollout answers is whether it fits the latency budget the old one set. We gave it a 25% p95 budget.

Setup. The 7B was already serving. We added the 9B as a second deployment on the same endpoint with no traffic and no replicas; the rollout starts it when it needs it.

# $MODEL_ID is Qwen/Qwen3.5-9B-FP8. --config is optional when the model has exactly one serving config. tg beta endpoints deploy $MODEL_ID --endpoint $ENDPOINT_ID --config $CONFIG_ID \ --min-replicas 0 --max-replicas 0 --traffic-weight 0

Start. One command creates and starts the canary: 10% → 50% → 100%, with a gate that compares the target's p95 router latency against the source's over a 5-minute window after each step.

tg beta endpoints rollout $TARGET_DEPLOYMENT_ID --source $SOURCE_DEPLOYMENT_ID \ --canary --steps 10,50,100 \ --metric router_latency --metric-stat p95 \ --metric-max-regression 25 --metric-direction higher-is-worse \ --metric-window 300s

What happened, by the clock (time since the rollout started):

"state": "ROLLOUT_STATE_SYSTEM_PAUSED", "currentTrafficPercent": 10, "pauseInfo": { "reason": "metrics-regression: regression check router_latency failed: target=1740.38 vs source=734.20, regression=137.05% exceeds max=25.00% (direction=REGRESSION_DIRECTION_HIGHER_IS_WORSE)", "pausedAt": "2026-09-16T20:50:45Z" }, "status": { "condition": { "category": "ROLLOUT_FAILURE_CATEGORY_METRIC_REGRESSION", "atStep": 0, "metrics": [ { "name": "router_latency", "stat": "METRIC_STAT_TYPE_PERCENTILE", "percentile": 95, "check": "METRIC_CHECK_TYPE_REGRESSION", "sourceValue": 734.20, "targetValue": 1740.38, "maxRegressionPercent": 25, "direction": "REGRESSION_DIRECTION_HIGHER_IS_WORSE", "verdict": "METRIC_VERDICT_BREACHED" } ] } }

Decide. The regression is real, not a blip: the 9B is a reasoning model and, at the same max_tokens, generates more per request. That is a product decision rather than something to resume past, so we canceled and went back.

tg beta endpoints rollout $ENDPOINT_ID --cancel --reason "p95 latency regression on the Qwen3.5 target" # split frozen at 90% / 10%; both deployments keep serving

tg beta endpoints rollout $SOURCE_DEPLOYMENT_ID --source $TARGET_DEPLOYMENT_ID --blue-green # the 7B takes 100% back, the 9B drains to zero

+11:09 CANCELED . The split froze at 90/10 within 0.2 s of the command. +17:00 the reverse rollout completed, 5 min 49 s after it started: 100% of traffic back on the 7B, the 9B drained to zero and stopped. Most of that time was a second 7B replica cold-starting, because after a cancel the default final replica count is the pair's combined count. The probe's verdict across the whole run, including the shift, the pause, the cancel and the reverse: 6,800 requests, 0 non-200 responses .

Audit trail. Every step above is in the endpoint's event feed, filterable by rollout ID:

tg beta endpoints events $ENDPOINT_ID --subject-id $ROLLOUT_ID --json

20:39:45 rollout.created canary rollout created: dep_src → dep_tgt, 3 step(s) to 100% 20:39:45 rollout.started rollout started: step 1 of 3 targets 10% traffic 20:43:42 rollout.traffic_shifted 0% → 10% target traffic: 0% → 10% 20:50:45 rollout.system_paused paused automatically at step 1 of 3: a metric check failed 20:50:54 rollout.canceled cancel requested; traffic will be frozen at the current split 20:50:54 rollout.canceled_complete canceled: traffic frozen at 90%/10% (source/target) An earlier run in July, with a deliberately impossible threshold gate, produced the same shape: a trip at 10% of traffic and 1,198 probe requests with zero errors through the recovery.

1. Two deployments on one endpoint. Keep your current deployment as the source and add the candidate as a target with zero traffic. pip install -U together (2.34.0 or newer) gives you the tg CLI:

# A stopped, zero-traffic target; the rollout restarts it when it scales it up. # Add --config cr_... to pin a specific config revision. tg beta endpoints deploy $MODEL --endpoint $ENDPOINT_ID \ --min-replicas 0 --max-replicas 0 --traffic-weight 0

2. Create and start a canary with the default ladder (5% → 25% → 50% → 100%) and one router_latency regression gate:

tg beta endpoints rollout $TARGET_DEPLOYMENT_ID --canary \ --metric router_latency --metric-stat p95 \ --metric-max-regression 10 --metric-direction higher-is-worse

3. Watch it with tg beta endpoints get $ENDPOINT_ID (or the endpoint's Rollouts tab in the console) as it progresses through the steps. Pause, promote or cancel it with tg beta endpoints rollout $ENDPOINT_ID --pause | --promote | --cancel .

Throughout, the endpoint URL and your clients stay unchanged; only the model behind them moves.

📚 Docs: Start a rollout · Gate rollouts with metrics · CLI reference · API reference: Create a rollout

‍