Evaluation scorecard
0 judged queriesThe answer key. A fixed list of test queries and, for each query, the listings that were graded as good. Every level is scored against the same answer key, so the numbers can be compared. in Chapter 11's fixed test split. Their normalized texts are excluded from tower training and validationExamples used to choose settings, not to fit the model or to report the final score. Chapter 11 keeps query groups separate: training fits the towers, validation selects the epoch, budget and score weight, and test queries only measure the result.. Each chapter is evaluated on the same subset. Chapter 8 was trained earlier on the full simulated log; this is a holdout for the new towers, not for every upstream model.
Grade one query row by row →The held-out split requires current Chapter 11 artifacts. Run npm run data:two-tower and restart the app.
Read the numbers
When a higher level scores lower
It happens, and the table shows it instead of hiding it. A chapter adds one rule. Hand-tuned boostsA hand-chosen number added to the score when a listing has a wanted property, for example the category the query points at. win on 'marble' and lose on queries where recency or price pushes the right listing down. That is the reason chapter 8 learns the weightsThe number a feature is multiplied by when the score is built. Chapter 7 picks these numbers by hand; chapter 8 learns them from clicks. from data, and the reason the introduction starts with an answer key: one convincing example is not proof. The average over all judged queriesThe answer key. A fixed list of test queries and, for each query, the listings that were graded as good. Every level is scored against the same answer key, so the numbers can be compared. is.
Ground-truth disclosure
Ground truth is the WANDS human relevance labels: Exact = 3, Partial = 1, Irrelevant = 0, over the evaluated queries. The labels come from human annotators, not clicks. The click log used by the learned and bias chapters is simulated from those grades times a position decay. Price and date are deterministic synthetic overlay columns because WANDS has neither.
Inspect the click log and the position decay →Chapter 11 evidence
The towers use training-query WANDS relevance grades. Separate validation queries select the epoch, candidate budget and score weight. Test query texts are excluded from those steps, but related paraphrases can cross the split. The labels are human; the clicks are simulated. Wider-hybrid and untrained title-vector controls are measured on validation queries in the model manifest.
Explore the evaluation stage in Chapter 11 →Provenance of chapter 8
- Training data
- simulated click sessions from the bundled log
- Clicks
- simulated, not observed
- Position bias
- correctedwith inverse propensity scoringInverse propensity scoring. Each click is divided by how often that position gets clicked, so a click far down the list counts for more than a click at the top. This removes part of the position effect from the training signal. (IPS)
- Rebuild
- npm run data:train