Both Folds Chose Zero: Fitting Rank Fusion Weights Instead of Guessing Them

If you have several retrieval branches — vector similarity, keyword matching, entity matching — you need some way to turn several ranked lists into one.

There are two standard answers. Give each branch a weight and add up the boosts, tuning the weights by hand until the results look good. Or use Reciprocal Rank Fusion, which combines by rank position rather than score and is widely treated as the safe default when you don't want to tune anything.

I had both. I replaced them with neither, and the reason is a number I did not expect.

The problem with hand-tuned weights

Hand tuning has a failure mode that's easy to miss from the inside.

You pick some weights. You run a handful of queries. You look at the results and they seem better, so you keep the weights. What you have actually done is fit parameters to the specific examples you happened to look at, using your own judgement as the loss function, with no holdout.

That's overfitting. It just doesn't feel like overfitting, because it's happening in your head rather than in a training loop. The tell is that you can never say how much better the weights are — only that they looked better on the queries you tried.

RRF avoids that by not having weights to tune, which is genuinely why people reach for it. But "no parameters to overfit" is not the same as "correct for your data." It's a prior, and priors can be wrong.

Fit them instead

The alternative is to treat fusion weights as what they are — parameters — and fit them properly, with a holdout so you can tell whether you've learned something or memorised something.

I had a labelled evaluation set: queries paired with the facts that should have come back. That makes this straightforward. Split the sets into folds, fit the weights on one fold, evaluate on the other, and see whether the fitted weights generalise.

192 labelled sets. Two-fold holdout.

Both folds, independently, chose a boost weight of zero.

Plain cosine similarity ranking beat the hand-tuned weights and beat RRF.

Why zero is a stronger result than a small number

If one fold had come back with a small weight and the other with a slightly different small weight, that's a normal result. You'd average them, ship it, and move on.

Zero is different, and two independent folds agreeing on zero is different again.

A fitted weight of zero says the optimiser was offered the extra signal and declined it — that including the other branches' boosts made held-out ranking worse, not better. And two folds landing there separately means it isn't an artifact of how the data happened to split.

The honest interpretation is that for this task, the additional branches were contributing more noise than signal to the ranking, and the cleanest thing to do with them was to stop letting them perturb the order.

That doesn't mean the other branches were useless in general — they still matter for getting candidates into the pool, which is a different job from ordering the pool. It means the fusion step specifically was worth nothing here, and the complexity it added was buying negative value.

What I'd want to know if I read this

I'll be straight about a gap: I have the fitted weights and the direction of the result, but I didn't record the size of the score gap between cosine, the tuned weights and RRF. That number would make this post considerably more useful, and its absence is a lesson of its own — write down the margin, not just the winner. A result you can't quantify later is a result you'll end up re-running.

What generalises

Fusion weights are parameters. Fit them. If you have a labelled set good enough to evaluate retrieval, it's good enough to fit a handful of fusion weights on, and doing so takes very little work compared to the tuning session it replaces.

Always hold out. The entire value of this exercise was being able to distinguish "these weights are right" from "these weights match the examples I looked at." Without a holdout you cannot tell those apart, and your intuition will confidently report the wrong one.

RRF is a default, not a law. It's a reasonable thing to reach for when you have nothing to measure against. Once you do have something to measure against, measure.

Complexity should have to earn its place. A fusion layer is code that has to be maintained, reasoned about and debugged. When held-out data says its optimal contribution is zero, that's permission to delete it — which is a better outcome than a weight you'd have kept tuning forever.

Two folds agreeing is worth more than one fold being confident. The cheapest way to find out whether you've learned something real is to learn it twice, separately.