How to Let an Agent Touch Production Without Letting It Delete Production

TL;DR: Do not ask whether an agent should have production access. Ask which single operations it should be able to perform, on which records, with which credential, under which approval, and with what way back. Read access, reversible writes, and irreversible deletes are three different risk classes and should never travel together in one token or one tool. Scope the credential to the job, require a dry run or a preview before a write, gate destructive actions behind a human who sees the actual change, keep backups that live outside the blast radius, log every call, and keep a kill switch that works when the agent is mid run. If any one of those pieces is missing, the agent is not ready for production, however good the demo looked.

Key Takeaways

• Production access is not binary. Split it into reads, reversible writes, and irreversible or destructive actions, and grant each class separately.
• The credential does the real governing. A careful prompt with a broad token is still a broad token.
• A dry run that shows the exact records and fields about to change is worth more than a generic approval prompt that a tired human will click through.
• Backups only count if they sit outside the system the agent can reach and you have actually restored from them.
• Log every tool call and keep a kill switch. Without those two, you cannot reconstruct an incident or stop one in progress.

Why is the usual question the wrong question?

Teams ask whether to give an agent production access, as if production were one room with one door. It is not. Reading a customer record, updating a non critical field, sending a message to a real user, and deleting a database are all production actions, and they differ by orders of magnitude in how much damage a mistake does and how hard it is to undo.

The incident pattern makes this concrete. In how an agent deletes a production database, the mechanism was ordinary: an authenticated call to a permitted endpoint using a credential that carried far more authority than the task needed. There was no exotic failure and no malicious model. The system simply allowed a routine looking step to become an irreversible one because the permissions made no distinction between them.

What should an agent be allowed to do by default?

Start from read only, and make every escalation earn its place.

First, reads. Let the agent look at production data when the job needs current data, but scope the reads too. Mask fields the job does not need, exclude secrets and unrelated customer records, and prefer a replica or an export over direct access to the primary store where that is practical. Failure mode: a read only token that can still dump an entire table, including data the task never required.

Second, reversible writes. These are changes you can undo quickly and provably: updating a status, adding a note, creating a draft, tagging a record. Grant these per tool and per resource type, not as a general write permission. Failure mode: a tool described as an update that can also overwrite fields outside its stated purpose because the underlying endpoint accepts anything.

Fourth, destructive and irreversible actions. Deletes, purges, migrations that drop data, credential changes, and permission changes should not be agent tools at all in most setups. If a workflow genuinely needs one, put a human in the execution path, not just in an approval queue, and require a separate credential for that step. Failure mode: delete authority inherited quietly because it shared a token with routine work.

What are the steps, in order?

1. Write the job in one sentence before you grant anything. Name the task, the records it touches, and the outcome that counts as done. If you cannot write that sentence, you cannot scope the permission. Failure mode: granting access for a vague brief like tidy up the database, then discovering the agent interpreted tidying broadly.

2. Issue a credential for that job only. Scope it by resource, by operation, and where your platform allows it, by time. Short lived credentials limit the damage from a leak and from a runaway run. Never hand an agent a personal admin token or a general purpose API key because it was convenient. Failure mode: the credential outlives the task and becomes a standing backdoor that nobody remembers creating.

3. Require a preview before any write. The agent should produce a dry run: what will change, in which records, from what value to what value, and how many records are affected. Show counts prominently, because an operation that touches far more records than expected is the clearest early warning there is. Failure mode: a preview in vague prose, such as updating some stale accounts, that gives the reviewer nothing to check.

4. Gate by consequence, not by habit. Low risk, reversible changes inside the agreed scope can run without a prompt. Anything destructive, outward facing, unusually large, or outside the preview needs a human decision on the specific change, not a blanket yes to the run. Design the gate against the failure described in approval fatigue and why human in the loop fails: if humans must approve everything, they soon approve anything, and the gate becomes decoration.

6. Put the backup outside the blast radius. A backup stored in the same account, on the same volume, or deletable with the same credential is a copy, not a safety net. Test restores on a schedule. An untested backup is a hope, and hope is not a control. Failure mode: discovering during an incident that the backup job failed quietly weeks earlier.

7. Log every call as if you will have to explain it. Record the agent identity, the tool, the arguments, the result, and the approval that authorised it. Logs are how you tell a bad instruction from a bad tool from a bad permission after the fact. Failure mode: logs that show a deletion happened but not which run requested it or what the preview said.

8. Keep a kill switch and rehearse it. Someone on the team should be able to revoke the credential and stop the run in one action, without needing the agent to cooperate. Rehearse it once before you need it. Failure mode: a kill switch that lives in the same console the incident is currently breaking.

What goes wrong most often?

The most common failure is scope that grows by accident. A tool built for one team gets reused by another, a temporary broad permission becomes permanent, a helper endpoint exposes more than its name suggests. Review agent permissions the way you review human admin access, regularly and with a bias toward removal.

The third failure is building the gate and then routing around it for speed. If the approval step is slow, fix the preview so decisions are fast. Do not solve friction by removing the one control that separates a mistake from an incident.

How do you know it is working?

You know it is working when you can answer four questions without guessing. Which credentials can the agent use right now, and what can each one do? What did the agent change in production this week, according to the logs rather than its own summary? How would you undo its largest recent change? And who can stop it, and how, at short notice?