A deployment can show a successful rollback while the agent's work continues to unfold.
The prior configuration is live again. But an in-flight run may still be executing. A ticket has already been closed, an order released, a customer message sent, or a record changed. Credentials issued to the workflow may remain valid. The dashboard says the release reverted; the business state did not.
Reverting an agent version is one recovery action. It does not automatically stop active work, reverse external effects, repair downstream state, preserve decision evidence, or prove that service can safely resume. A production agent needs a rollback-and-reconciliation design that treats those as separate responsibilities.
Rollback has entered the deployment vocabulary
OpenAI's July 22 Presence announcement describes a managed enterprise deployment in which teams can test a proposed change against the production version before approving a controlled rollout. The product combines that change process with policies, guardrails, approved actions, simulations, evaluation tools, and escalation rules.
Rollback is becoming part of the expected operating surface for agents, not an improvised engineering maneuver after a bad release. The word can still hide different actions: routing new traffic to an earlier configuration, pausing new invocations, terminating active sessions, disabling a tool, revoking an identity, reversing a transaction, moving work to a human queue, or restoring a clean data snapshot.
Those actions touch different systems and carry different business costs. A production design should say which one the control performs.
A version boundary is not an effect boundary
Microsoft Foundry documentation makes one boundary unusually clear. A model update applies to new conversations, while in-progress conversations continue with the previous model until they complete. Restoring a prior version rolls back the agent configuration; evaluation runs and deployment records remain in project history.
That behavior belongs to a particular product path, not every agent platform. It exposes the question every deployment should answer: which work receives the version decision, and what happens to work that crossed the boundary earlier?
Amazon Bedrock documentation describes immutable agent versions and aliases that can point to a prior version. The alias control can also reject new invocations without an IAM-policy change. An operator therefore has at least two distinct levers: change which version receives traffic, or pause new calls.
Neither description says that a version switch reverses an external action. Once an agent has used a tool, the system of record becomes part of recovery.
Separate four recovery planes
Use four different verbs. Revert the control configuration. Contain active and new execution. Reconcile the effects already written to business systems. Resume only after the evidence supports another operating decision.
My proposed operating model maps those decisions across four planes:
| Plane | Recovery question |
|---|---|
| Control | Which prompt, model, policy, tool definition, connector, and routing configuration is now active? |
| Execution | Which sessions, runs, jobs, queues, credentials, and approvals were active during the exposure window, and can they be paused or allowed to finish safely? |
| Business state | Which external records or transactions changed, which effects are reversible, and which require a compensating action or human decision? |
| Evidence | Which version, identity, input, tool call, approval, result, and state transition must remain available to reconstruct the event and authorize resumption? |
A control-plane rollback may be complete while the other three planes remain open. That is not a failed platform feature. It is an unfinished recovery design.
The distinction also prevents destructive reflexes. Stopping every session could strand work between systems. Revoking a shared credential could disable unrelated workflows. Restoring a database snapshot could erase legitimate transactions completed after the bad release. Components can have different owners, recovery methods, and acceptable failure costs.
Write the rollback-and-reconciliation record
For one consequential workflow, write this record before production:
- Trigger: the signal and decision threshold for freezing rollout, rejecting new work, or invoking a prior version. Prove it with a routed test signal and recorded decision.
- Exposure window: the first possibly affected invocation, last accepted invocation, and versions, identities, tools, queues, and systems involved. Prove it with trace-to-system correlation.
- Containment sequence: the ordered controls for traffic, active execution, credentials, connectors, and human fallback. Prove each control independently before testing the sequence.
- Effect inventory: the authoritative query that finds every external record or transaction changed during the window. Test it against known fixtures and omissions.
- Disposition: the owner and permitted response for each effect: retain, reverse, compensate, escalate, or leave untouched with an accepted reason.
- Evidence hold: the records that must not be overwritten during reconciliation, with access and retention owners.
- Resumption gate: the regression checks, reconciled-state evidence, operating approval, and graduated authority required before new work enters.
This is Scott's proposed recovery record, not a description of any source company's internal procedure.
A compensating action should also have authorization and idempotency checks. Running it twice should not create a second refund, duplicate message, or another unintended mutation. The resumption gate needs a named person who can accept the remaining business risk.
Test the containment cost before the incident
An AWS incident-response presentation hosted by NIST warns that improvised containment can corrupt downstream state or break dependent services. It recommends decision trees for each AI component, tied to the business function, containment options, cascading effects, business cost, and authority to accept that cost.
The fastest way to stop an agent may not be the safest way to preserve the workflow it supports.
Run a recovery game day with authorized test data. Route new work to the prior version. Reject another invocation. Let one safe in-flight run finish and stop another. Expire the execution credential. Find every mutated record in the test window. Apply a compensating action twice and confirm the second attempt creates no duplicate effect. Preserve the trace, then make the permanent operator decide whether service may resume.
NIST's AI Risk Management Framework connects safe decommissioning with response, recovery, override, incident management, and change management. It also calls for assigned responsibility to supersede, disengage, or deactivate systems whose performance or outcomes conflict with intended use. Recovery is broader than selecting an older artifact. It is a governed return to acceptable operation.
Put reversibility in the FDE handoff
The handoff should identify version and routing controls, active-session behavior, execution identities, tool side effects, authoritative effect queries, compensating actions, evidence-retention rules, the fallback path, and the resumption owner. It should state plainly which effects cannot be reversed.
Then the permanent team should run the game day while the FDE observes. If the original builder must identify affected records, operate the pause, or decide that the remaining state is acceptable, the capability has not transferred.
Version rollback is necessary infrastructure. It is not a complete recovery plan.
Before increasing an agent's authority, ask the operator to point to the last known good version, stop new work, account for every in-flight run, reconcile every exposed effect, and produce the evidence for resumption. If the team can do only the first step, it can roll back the agent. It cannot yet recover the workflow.