Tune ranking weights without a deploy
Your relevance weights are magic numbers buried in a scoring function. Every tweak is a pull request, a review, and a release — and still nobody measures what it did to conversion.
The problem
Your relevance weights are magic numbers buried in a scoring function.
Ranking is the highest-leverage code in a lot of products and the least experimented on, because the loop is so slow. A weight change takes a day to ship and a week to argue about, so the team tunes it twice a year instead of twice a sprint.
It is also the easiest place to fool yourself. Relevance improvements often trade against latency or result diversity, and if you only watch click-through you will happily ship something that makes search feel worse.
What the team needs is not a machine-learning platform. It is the ability to move four numbers in production, on a slice of traffic, with latency and zero-result rate watched automatically.
# before — a deploy per experiment def score = ( 0.55 * relevance + 0.20 * recency - 0.10 * same_seller_penalty ) # after — the same numbers, resolved # per request from one parameter set
What you put under control
Declared once, then governed centrally. This is the only code step.
| Parameter | Type | Default | Range |
|---|---|---|---|
| ranking.relevance_weight | number | 0.55 | 0–1 |
| ranking.recency_boost | number | 0.20 | 0–1 |
| ranking.diversity_penalty | number | 0.10 | 0–1 |
| ranking.candidate_depth | number | 200 | 50–1000 |
| Surface | Kind | Role |
|---|---|---|
| Search results | ranking | decision |
| Category pages | ranking | decision |
| Recommendations | ranking | decision |
Those bindings are what compute the risk class — high: ranking surface · decision binding.
| Variant | relevance_weight | recency_boost |
|---|---|---|
| control | 0.55 | 0.20 |
| relevance-led | 0.68 | 0.12 |
| freshness-led | 0.48 | 0.34 |
Risk class: high
ranking surface · decision binding. A human approves the measurement plan before any traffic, and again before the value becomes the product default.
How it runs
The mechanism is configuration. The application code is already done.
A high-risk surface never starts traffic without a human approving the plan. The experiment establishes causality across the three arms; latency and zero-result rate are blocking guardrails throughout.
Because relevance preferences differ by user, a linear contextual bandit can then personalize the weights using the context you already pass in — no model to deploy.
What you get back
A verdict you can defend, and the reasoning kept for next time.
Four numbers you can finally move, with the two things that usually get traded away — latency and zero-result rate — watched on every phase rather than checked afterwards.
When it completes, the winning values become the parameter defaults and the whole thing — question, variants, evidence, verdict — is retained. The next person to propose moving this number starts from the answer instead of the argument.
The question asked, the values tried, the interval at the moment of the call, which thresholds were in force, and who approved the promotion.
Keep reading
The mechanics behind this page.
Start with this one number
Parameterize it this week. The first measured change usually pays for the integration.