Read this after the chapter 8 lesson.Open chapter 8 →
A ranker trained on raw clicks learns to copy the ranking that produced them.
Searchers click what they see: the top result gets most of the clicks, whatever its quality. A model that counts every click as a vote therefore rewards whatever was already on top. That is position biasSearchers click the top results more often whatever their quality. A model that trusts raw clicks therefore learns to copy the old ranking, mistakes included.. The fix: multiply each click by how likely its position was to be seen, so a click low on the page weighs more. That is inverse propensity scoringInverse propensity scoring. Each click is divided by how often that position gets clicked, so a click far down the list counts for more than a click at the top. This removes part of the position effect from the training signal..
- How this click log was made
- Sessions
- 50000 simulated searchers on 246 queries
- Who clicked
- Each searcher clicked a shown listing with a probability set by its WANDS grade times a position-decay factor. A weakly relevant listing at position 1 still earned clicks, which is how the bias enters the log.
- Clicks
- 139310 clicks in total. Result 1 was clicked by 70% of sessions, result 10 by 6%.
- Real click logs look like this: relevance and position mixed together. The clicks come from the searchers; the grades come from a separate answer key. A model cannot tell the two apart from a click alone. That is the problem this page fixes.
Do not compare these numbers with the scorecard. This page trains two small models on the click log above and scores them on the book's 246 teaching queries. The scorecard grades the full chapter 8 pipeline on its own answer key. Same metric, different test. The two models differ in one thing only: how they treat a click. The lab below shows which listings move.
Click rate by result position
The corrected model divides each click by this click rate, so a click at position 10 counts for more than a click at position 1.
Useless listings searchers clicked
8 cases- wainscoting ideasmarkovich 5 drawer combo dresser with mirror#113/203
- industrialalameda end table#112/203
- pull out sleeper loveseatefim 84 '' wide reversible sleeper sofa & chaise#111/203
- 48 in entry table with side by side draweramisha 1 - drawer nightstand in light brown#110/204
- blk 18x18 seat cushionsergonomic task chair#110/204
- living room designsstapp 2 piece faux leather living room set#110/203
- barstool patio setsaiman 3 piece bistro set#210/204
- tall storage cabinetarbyrd armoire#210/203
Every row is a listing the answer key grades 0 (useless) that searchers still clicked. It was shown near the top, so it was seen and clicked. Each click enters training as a vote for relevance, so the model learns to rank it higher. That is the bias, in the raw data.
Run the two models on the same log
Same log, two models
chapter 8 lab- 1doty mid century modern upholstered 3 piece living room setgrade 3#297/203
- 2bethnal living room setgrade 0weighting drops it 3#31/203
- 3stapp 2 piece faux leather living room setgrade 0weighting lifts it 1#110/203
- 4stcyr 3 piece faux leather living room setgrade 0weighting lifts it 1#42/203
- 5boadicea mid century 2 piece living room setgrade 0weighting lifts it 1#51/203
- 6macaire 45 '' wide club chairgrade 0#70/203
- 7seneca 2 piece standard living room setgrade 0#81/203
- 8deven configurable living room setgrade 3weighting drops it 1#913/203
- 9winston 16 '' wide square floral footstool ottomangrade 0weighting lifts it 1#101/203
- 10caudell mid-century modern fabric loveseatgrade 0#62/203
Sorted by the raw-click model. One grade-0 listing ranks higher here than under weighting (shaded). Switch the sort to see them drop.
What one click is worth
A click only proves the searcher looked. Divide by how likely that position was to be seen, and a click far down the page outweighs one at the top.
Position 1 is examined with probability 1.00, so one click there is worth 1.0 clicks at the top. The simulator's curve is the 1/rank default; drag to see how a slower or faster fade changes the weight. Training caps the weight at 8, so one click never counts for more. This slider only moves the bars above; the table on the left always uses the log's estimated curve.
The correction, named
Inverse propensity scoringInverse propensity scoring. Each click is divided by how often that position gets clicked, so a click far down the list counts for more than a click at the top. This removes part of the position effect from the training signal.: each training pair is weighted by 1 / estimated P(click | rank), used as a proxy for P(examine | rank).
Why the uncorrected log copies the ranking
The log was produced by a ranker. Clicks concentrate at the top regardless of relevance, so "was clicked" mostly means "was shown first". A model fit to that signal learns the old ranker's preferences, and its mistakes become training data.
In your own service: if your only click data comes from the ranking you already have, treat its order as suspect. A random slice of results is the clean way to measure position.