Nobody could say which prompt was in production

The first incident review on a new agent often starts with a simple question that nobody can answer. Which version of the prompt produced this reply? The prompt had been edited in a web console, a colleague had tweaked it the week before, and the model name pointed to an alias the provider updates on its own.

“Which version answered this?”

An agent is more than a prompt. The behavior comes from the prompt, the model, the retrieval settings, the tools and their descriptions, and the guardrail rules together. We treat that set as one versioned bundle. Every trace records the bundle version, so any answer can be tied to the exact configuration that gave it.

“Why is the prompt not in the repository?”

Prompts are code in everything but name. We keep them in version control, review changes like any other change, and tag a release. The edit that lived only in a console is the one that cannot be explained three weeks later.

“What happens when the provider updates the model?”

We pin model versions instead of following an alias. When a new version is available, we run the evaluation set against it first and compare groundedness, retrieval-dependent answers and cost. We switch when the numbers support it, and we keep the previous version ready for a rollback.

“How do we roll out without betting everything?”

We release in stages. A new bundle first runs in shadow mode on live traffic without affecting users, then reaches a small share of requests, and then widens if the metrics hold. If they do not, the rollback is one configuration change.

“Who signed this off?”

Each release carries its evaluation results and the name of the person who approved it. When an auditor or a client asks why the agent behaved as it did on a given date, the answer is in the change log and not in someone's memory.