Test an AI change like a product change
Temperature, system prompt, model choice, token ceiling — all hardcoded, all iterated by redeploying, and all evaluated by how the output felt to whoever was looking.
The problem
Temperature, system prompt, model choice, token ceiling — all hardcoded, all iterated by redeploying, and all evaluated by how the output felt to whoever was looking.
AI features moved fast enough that nobody built the measurement discipline around them. Prompts get rewritten in a pull request, a model gets swapped because a newer one shipped, and the evidence that any of it helped is a handful of screenshots in Slack.
The costs are real and easy to miss. A prompt that improves answer quality can double token spend. A cheaper model can quietly raise escalations to a human. Neither shows up unless you are watching cost and deflection alongside quality.
A model swap is a change to your product. It deserves a canary, a control, a guardrail on cost, and a record of why you kept it.
// before const MODEL = "sonnet-4" TEMPERATURE = 0.7 SYSTEM_PROMPT = promptV3 MAX_TOKENS = 1024 // after — four parameters, one // change, a real control arm
What you put under control
Declared once, then governed centrally. This is the only code step.
| Parameter | Type | Default | Range |
|---|---|---|---|
| ai.assistant.model | string | "sonnet-4" | enum |
| ai.assistant.temperature | number | 0.7 | 0–2 |
| ai.assistant.prompt_version | string | "v3" | enum |
| ai.assistant.max_tokens | number | 1024 | 128–4096 |
| Surface | Kind | Role |
|---|---|---|
| In-app assistant | ai | decision |
| Support deflection | ai | decision |
| Email drafting | ai | side_effect |
Those bindings are what compute the risk class — high: AI surface · side-effect binding.
| Variant | prompt_version | temperature |
|---|---|---|
| control | "v3" | 0.7 |
| tighter-prompt | "v4" | 0.4 |
| new-model | "v3" | 0.7 |
Risk class: high
AI surface · side-effect binding. A human approves the measurement plan before any traffic, and again before the value becomes the product default.
How it runs
The mechanism is configuration. The application code is already done.
AI surfaces compute to high risk, so a human approves the measurement plan before any traffic and again before promotion. The canary catches the catastrophic cases — runaway tokens, refusal loops — on 1% of sessions.
Prompt variants are a natural fit for adaptive optimization once you trust the guardrails, since the best phrasing often differs by user intent.
What you get back
A verdict you can defend, and the reasoning kept for next time.
Prompt and model decisions with evidence attached — and a cost guardrail that stops a clever prompt from tripling your inference bill.
When it completes, the winning values become the parameter defaults and the whole thing — question, variants, evidence, verdict — is retained. The next person to propose moving this number starts from the answer instead of the argument.
The question asked, the values tried, the interval at the moment of the call, which thresholds were in force, and who approved the promotion.
Keep reading
The mechanics behind this page.
Start with this one number
Parameterize it this week. The first measured change usually pays for the integration.