Evaluation scorecard
Every chapter that runs, graded against the same answer key of 246 judged queriesThe answer key. A fixed list of test queries and, for each query, the listings that were graded as good. Every level is scored against the same answer key, so the numbers can be compared.. This overall view includes Chapter 11 training and validation queriesExamples used to choose settings, not to fit the model or to report the final score. Chapter 11 keeps query groups separate: training fits the towers, validation selects the epoch, budget and score weight, and test queries only measure the result.. Use the held-outExamples excluded from training. Chapter 11 reserves normalized query groups before creating training pairs. Validation groups select an epoch; test groups only measure the frozen model. view to assess its new towers.
Grade one query row by row →Where each chapter helps
Read this table first. Each row is the trap one chapter is built to fix. The numbers are still real NDCG@10, not a demo. Scroll right for the later chapters.
| Query family | Ch. 1 | Ch. 2 | Ch. 3 | Ch. 4 | Ch. 5 | Ch. 6 | Ch. 7 | Ch. 8 |
|---|---|---|---|---|---|---|---|---|
| Typos and paraphrases4 queries. Chapters 1 to 3 cannot match these words. Chapter 4 corrects spelling and chapter 5 matches attributes; a paraphrase with no shared words still needs chapter 9's vectors. | 0.000 | 0.000+0.000 | 0.000+0.000 | 0.250+0.250 | 0.287+0.037 | 0.287+0.000 | 0.304+0.017 | 0.258−0.046 |
| Ambiguous category2 queries. The same word sits on the product and on an accessory, or in two categories. Chapter 7's category boost should lift the intended listing. | 0.378 | 0.339−0.040 | 0.620+0.282 | 0.620+0.000 | 0.620+0.000 | 0.620+0.000 | 0.575−0.046 | 0.772+0.198 |
| Typical queries240 queries. Short catalog queries with no hard case. Later chapters should not make these much worse. | 0.399 | 0.428+0.029 | 0.545+0.116 | 0.543−0.001 | 0.575+0.032 | 0.578+0.003 | 0.582+0.004 | 0.582+0.000 |
Average over the selected query set
One number per chapter. Easy queries and hard queries are mixed, so a real win on a small family can disappear here.
| Level | NDCG@10A score from 0 to 1 for the first 10 results. It is 1.0 when the best listings are at the top. A good listing low in the list, or a poor listing high in the list, lowers it. Short for normalised discounted cumulative gain. | vs prev | MRRMean reciprocal rank. It looks only at the first good listing: position 1 scores 1, position 2 scores 1/2, position 5 scores 1/5. The average over all test queries. | Recall@50The share of all good listings that were found at all (here: in the first 50). Recall asks: did we find them? | p50 ms | p95 ms |
|---|---|---|---|---|---|---|
| 1Recency search | 0.393 | — | 0.645 | 0.091 | < 0.1 | 2.0 |
| 2Query normalization | 0.421 | +0.028 | 0.700 | 0.104 | 2.0 | 14.0 |
| 3Lexical match | 0.536 | +0.116 | 0.796 | 0.114 | 4.0 | 28.0 |
| 4Spelling normalization | 0.539 | +0.003 | 0.800 | 0.115 | 7.0 | 39.0 |
| 5Entity-tagged matching | 0.571 | +0.032 | 0.853 | 0.147 | 371.0 | 673.0 |
| 6Canonical search policy | 0.574 | +0.003 | 0.851 | 0.148 | 370.0 | 722.0 |
| 7Attribute boosting | 0.577 | +0.003 | 0.858 | 0.148 | 367.0 | 753.0 |
| 8Learned ranking | 0.578 | +0.001 | 0.857 | 0.149 | 366.0 | 657.0 |
Recall@50 measures only the first 50 results. Both a wider candidate set and a different top-50 order can change it.Query encoding excluded. Chapter 10 reuses Chapter 9 scores, so its time is incremental, not standalone latency.
Per-query regressions
180 queries- LevelQueryNDCG ΔOpen
- 3Lexical match3 piece rug set with runners−0.153compare →
- 5Entity-tagged matching3 piece rug set with runners−0.353compare →
- 7Attribute boosting3 piece rug set with runners−0.048compare →
- 3Lexical match70s inspired furniture−0.026compare →
- 7Attribute boosting70s inspired furniture−0.002compare →
- 7Attribute boostingabove toilet cabinet−0.025compare →
- 7Attribute boostingacrylic clear chair−0.064compare →
- 8Learned rankingacrylic clear chair−0.066compare →
- 7Attribute boostinganchor decor−0.091compare →
- 8Learned rankingbar room wall decor−0.038compare →
- 8Learned rankingbar stool 24 inches height−0.054compare →
- 5Entity-tagged matchingbar stool with backrest−0.155compare →
- 3Lexical matchbarstool patio sets−0.065compare →
- 2Query normalizationbathroom lighting−0.503compare →
- 7Attribute boostingbathroom single faucet−0.107compare →
- 5Entity-tagged matchingbathroom vanity knobs−0.397compare →
- 2Query normalizationbeaded curtains−0.267compare →
- 8Learned rankingbeaded curtains−0.067compare →
- 2Query normalizationbed risers−0.057compare →
- 7Attribute boostingbeds that have leds−0.009compare →
- 8Learned rankingberenice 4 storage cabinet−0.017compare →
- 7Attribute boostingblk 18x18 seat cushions−0.014compare →
- 7Attribute boostingbrush nickel faucet bathroom−0.003compare →
- 8Learned rankingbrush nickel faucet bathroom−0.097compare →
- 8Learned rankingburnt orange curtains−0.187compare →
- 3Lexical matchcake cutlery−0.028compare →
- 7Attribute boostingcake cutlery−0.003compare →
- 8Learned rankingcake cutlery−0.004compare →
- 2Query normalizationchaise lounge couch−0.302compare →
- 5Entity-tagged matchingchaise lounge couch−0.069compare →
- 8Learned rankingchaise lounge couch−0.023compare →
- 3Lexical matchclear shower curtain−0.031compare →
- 7Attribute boostingclear shower curtain−0.057compare →
- 8Learned rankingclear shower curtain−0.003compare →
- 8Learned rankingcloud modular sectional−0.111compare →
- 2Query normalizationcoffee container−0.044compare →
- 7Attribute boostingcoffee container−0.356compare →
- 7Attribute boostingcoffee table fire pit−0.060compare →
- 8Learned rankingcoffee table fire pit−0.186compare →
- 8Learned rankingcoma inducer comforter king set−0.008compare →
- 7Attribute boostingcover set for outdoor furniture−0.218compare →
- 8Learned rankingcover set for outdoor furniture−0.213compare →
- 3Lexical matchdecorative wall clocks−0.409compare →
- 7Attribute boostingdecorative white pillow−0.007compare →
- 3Lexical matchdesk and chair set−0.010compare →
- 8Learned rankingdesk and chair set−0.081compare →
- 2Query normalizationdesk for kids−0.026compare →
- 7Attribute boostingdesk for kids−0.003compare →
- 3Lexical matchdining room tables−0.118compare →
- 7Attribute boostingdining table vinyl cloth−0.163compare →
- 8Learned rankingdining table vinyl cloth−0.009compare →
- 7Attribute boostingdonaldson teak couch−0.452compare →
- 8Learned rankingdriftwood mirror−0.057compare →
- 3Lexical matchdrum picture−0.109compare →
- 8Learned rankingend table between recliners−0.012compare →
- 3Lexical matchentertainment console−0.026compare →
- 7Attribute boostingentertainment console−0.002compare →
- 8Learned rankingentertainment stand end table−0.002compare →
- 7Attribute boostingentrance table−0.001compare →
- 2Query normalizationergonomic chair−0.169compare →
- 8Learned rankingfarmhouse cabinet−0.051compare →
- 8Learned rankingfleur de lis living candle wall sconce bronze−0.024compare →
- 8Learned rankingfloating bed−0.023compare →
- 3Lexical matchfold in half table−0.073compare →
- 8Learned rankingfold in half table−0.189compare →
- 7Attribute boostingfront door cabinet with doors−0.041compare →
- 8Learned rankingfull bed with trundle−0.144compare →
- 4Spelling normalizationgeralyn upholstered storage platform bed−0.404compare →
- 8Learned rankinggold−0.003compare →
- 7Attribute boostinggray dresser−0.055compare →
- 8Learned rankinggray dresser−0.041compare →
- 7Attribute boostinghardwood beds−0.006compare →
- 8Learned rankinghardwood beds−0.018compare →
- 8Learned rankinghigh weight capacity bunk beds−0.051compare →
- 7Attribute boostinghome sweet home sign−0.011compare →
- 8Learned rankinghome sweet home sign−0.052compare →
- 7Attribute boostinghuntsville sectional sofa−0.059compare →
- 2Query normalizationindoor multi tiered plant stand−0.058compare →
- 2Query normalizationindustrial−0.079compare →
- 7Attribute boostingindustrial−0.091compare →
- 7Attribute boostingindustrial pipe dining table−0.018compare →
- 8Learned rankingking adjustable bed including mattress−0.164compare →
- 8Learned rankingkitchen islands with seating−0.038compare →
- 7Attribute boostingkitchen storage cabinet−0.014compare →
- 8Learned rankingkitchen storage cabinet−0.051compare →
- 3Lexical matchkitchen wooden stand−0.001compare →
- 5Entity-tagged matchingkitchen wooden stand−0.001compare →
- 7Attribute boostingkitchen wooden stand−0.001compare →
- 8Learned rankingkohler purist brushed bronze−0.003compare →
- 3Lexical matchkohler whitehaven farmhouse kitchen sink−0.111compare →
- 8Learned rankinglarge cushion outdoor chair−0.050compare →
- 3Lexical matchleather chair−0.151compare →
- 6Canonical search policyleather chair−0.015compare →
- 8Learned rankingleather chair−0.017compare →
- 8Learned rankingled nightstand−0.067compare →
- 3Lexical matchlight and navy blue decorative pillow−0.029compare →
- 8Learned rankinglight and navy blue decorative pillow−0.130compare →
- 8Learned rankingligth bulb−0.078compare →
- 7Attribute boostingliving room coffee table sets−0.016compare →
- 2Query normalizationliving room designs−0.208compare →
- 8Learned rankingliving room designs−0.064compare →
- 7Attribute boostinglove seat wide faux leather tuxedo arm sofa−0.004compare →
- 7Attribute boostingmattress foam topper queen−0.064compare →
- 2Query normalizationmeade mirror−0.086compare →
- 2Query normalizationmeditation−0.152compare →
- 2Query normalizationmedium clips−0.209compare →
- 7Attribute boostingmilk cow chair−0.004compare →
- 7Attribute boostingmodern outdoor furniture−0.017compare →
- 3Lexical matchmulti color rug−0.057compare →
- 5Entity-tagged matchingmulti color rug−0.137compare →
- 6Canonical search policymulti color rug−0.003compare →
- 2Query normalizationnesting tray set−0.182compare →
- 8Learned rankingodum velvet−0.035compare →
- 8Learned rankingorren ellis l shape desk−0.098compare →
- 3Lexical matchoutdoor light fixtures−0.038compare →
- 8Learned rankingoutdoor light fixtures−0.221compare →
- 8Learned rankingoutdoor lounge chair−0.016compare →
- 3Lexical matchoutdoor privacy wall−0.003compare →
- 8Learned rankingoutdoor privacy wall−0.029compare →
- 7Attribute boostingoutdoor sectional dining−0.017compare →
- 7Attribute boostingpapasan chair frame only−0.004compare →
- 8Learned rankingpapasan chair frame only−0.003compare →
- 2Query normalizationparsons chairs−0.018compare →
- 8Learned rankingpatio design−0.141compare →
- 8Learned rankingpedistole sink−0.105compare →
- 8Learned rankingpool floats−0.073compare →
- 6Canonical search policyqueen wingback chair−0.008compare →
- 3Lexical matchreal touch roses floral arrangements in glass vase−0.006compare →
- 5Entity-tagged matchingreal touch roses floral arrangements in glass vase−0.136compare →
- 8Learned rankingreal touch roses floral arrangements in glass vase−0.008compare →
- 8Learned rankingruckus chair−0.012compare →
- 8Learned rankingsalon chair−0.006compare →
- 7Attribute boostingseat cushions desk−0.124compare →
- 3Lexical matchsheets for twinxl−0.056compare →
- 7Attribute boostingshoe bench entryway−0.088compare →
- 8Learned rankingshoe bench entryway−0.004compare →
- 8Learned rankingshoe closet−0.005compare →
- 7Attribute boostingshower hooks−0.095compare →
- 8Learned rankingshower hooks−0.195compare →
- 7Attribute boostingside table with storage−0.007compare →
- 8Learned rankingsmall curtain rods−0.179compare →
- 5Entity-tagged matchingsmall space dining table and chairs sets−0.176compare →
- 8Learned rankingsmall wardrobe grey−0.014compare →
- 8Learned rankingsmart coffee table−0.095compare →
- 8Learned rankingsofa with ottoman−0.002compare →
- 3Lexical matchstainless steel colander set−0.103compare →
- 8Learned rankingstorage dresser−0.111compare →
- 2Query normalizationsunflower−0.095compare →
- 7Attribute boostingsuper bright lights−0.024compare →
- 7Attribute boostingtall storage cabinet−0.001compare →
- 6Canonical search policyteal chair−0.002compare →
- 7Attribute boostingtoddler couch fold out−0.006compare →
- 2Query normalizationtufted chair with gold legs−0.001compare →
- 3Lexical matchtufted chair with gold legs−0.187compare →
- 8Learned rankingtufted chair with gold legs−0.087compare →
- 2Query normalizationtufted upholstered bed diamond−0.153compare →
- 6Canonical search policyturquoise chair−0.034compare →
- 8Learned rankingturquoise pillows−0.005compare →
- 8Learned rankingtwin bed frame−0.227compare →
- 2Query normalizationunique coffee tables−0.159compare →
- 3Lexical matchunique coffee tables−0.004compare →
- 7Attribute boostingupholstered bed−0.005compare →
- 3Lexical matchvelvet chaise−0.144compare →
- 8Learned rankingvelvet chaise−0.037compare →
- 8Learned rankingwall shelves−0.069compare →
- 2Query normalizationwayfair sleep zippered−0.071compare →
- 7Attribute boostingwelcome sign−0.065compare →
- 8Learned rankingwelcome sign−0.055compare →
- 2Query normalizationwhimsical solar lights−0.081compare →
- 7Attribute boostingwindow wall accent−0.016compare →
- 8Learned rankingwine bar−0.081compare →
- 8Learned rankingwishbone chair−0.069compare →
- 2Query normalizationwood bar stools−0.043compare →
- 3Lexical matchwood bar stools−0.010compare →
- 2Query normalizationwood coffee table set by storage−0.018compare →
- 7Attribute boostingwood floor protector mat−0.035compare →
- 8Learned rankingwood floor protector mat−0.022compare →
- 5Entity-tagged matchingwooden chair outdoor−0.047compare →
- 8Learned rankingwooden chair outdoor−0.011compare →
- 8Learned rankingzen−0.004compare →
Read the numbers
NDCG@10 rewards putting the most relevant listings at the top of the first page. It is the headline metric because position matters to a searcher.MRR asks how far down the first relevant listing sits.Recall@50 measures the share of known relevant listings in the first 50. It can change through retrieval or through ordering when more than 50 candidates exist.
When a higher level scores lower
It happens, and the table shows it instead of hiding it. A chapter adds one rule. Hand-tuned boostsA hand-chosen number added to the score when a listing has a wanted property, for example the category the query points at. win on 'marble' and lose on queries where recency or price pushes the right listing down. That is the reason chapter 8 learns the weightsThe number a feature is multiplied by when the score is built. Chapter 7 picks these numbers by hand; chapter 8 learns them from clicks. from data, and the reason the introduction starts with an answer key: one convincing example is not proof. The average over all judged queriesThe answer key. A fixed list of test queries and, for each query, the listings that were graded as good. Every level is scored against the same answer key, so the numbers can be compared. is.
Ground-truth disclosure
Ground truth is the WANDS human relevance labels: Exact = 3, Partial = 1, Irrelevant = 0, over the evaluated queries. The labels come from human annotators, not clicks. The click log used by the learned and bias chapters is simulated from those grades times a position decay. Price and date are deterministic synthetic overlay columns because WANDS has neither.
Inspect the click log and the position decay →Chapter 11 evidence
The towers use training-query WANDS relevance grades. Separate validation queries select the epoch, candidate budget and score weight. Test query texts are excluded from those steps, but related paraphrases can cross the split. The labels are human; the clicks are simulated. Wider-hybrid and untrained title-vector controls are measured on validation queries in the model manifest.
Explore the evaluation stage in Chapter 11 →Provenance of chapter 8
- Training data
- simulated click sessions from the bundled log
- Clicks
- simulated, not observed
- Position bias
- correctedwith inverse propensity scoringInverse propensity scoring. Each click is divided by how often that position gets clicked, so a click far down the list counts for more than a click at the top. This removes part of the position effect from the training signal. (IPS)
- Rebuild
- npm run data:train