Search Evolution Explorer

Level 3 · Lexical match

src/domain/levels/level-1-lexical.ts · 72 lines
Back to search

Order by text fit, not by age

Keep the chapter 2 candidatesAll listings that matched the query at all, before they are ordered. Scoring can only reorder this set. It cannot add to it. unchanged. Then order them by how strongly their text matches: rare query words count most, repeated words count with diminishing returns, and long listings are normalisedRewrite a word into a standard form before looking it up, for example cut the ending or collapse 'mid century' to 'midcentury'. by length.

Where to look: match is passed through unchanged from chapter 2. score replaces the recency base with text fit plus a small, weighted freshness bonus.

Guided example

sofavs chapter 2

BM25 reorders the chapter 2 candidates by text fit instead of age: the sofa-and-chaise with the strongest word match moves above fresher but weaker listings.

Run it hereOpen in compare

Prerequisites

Reuses the ranking fromChapter 2
Foundation · Inverted indexbuilt at start
Declared artifactsready
src/domain/levels/level-1-lexical.ts72 linessame module the pipeline ran

Keep the chapter 2 candidatesAll listings that matched the query at all, before they are ordered. Scoring can only reorder this set. It cannot add to it., then order them by how well the words fit. MatchingThe step that decides which listings are candidates at all. passes through; scoringThe step that decides the order of the candidates. is new.

import { level1Content } from "../../content/levels/level-1.ts";
import { , } from "../index/bm25.ts";
import { } from "../signals/build-signals.ts";
import { , } from "../signals/freshness.ts";
import { } from "../text/analyze-document.ts";
import type { EvolutionLevel, LevelDeps, LevelRanker } from "../types/level.ts";
import { as } from "./level-1-normalization.ts";
/** How much a listing posted on the snapshot day can gain, in text-fit units (0 to 1). */
const FRESHNESS_WEIGHT = 0.1;
/** After this many days, the freshness bonus is half as large. */
const FRESHNESS_HALF_LIFE_DAYS = 7;
/** Duplication across levels is accepted so each evolution level stays independently readable. */
export function createRanker(deps: LevelDeps): LevelRanker {
const previous = (deps);
// The catalog is a fixed snapshot, so age is counted from its newest listing, not from the
// wall clock. The same evaluation then gives the same numbers on any day it runs.
const snapshot = (deps.index.availableItems);
return {
match: previous.match,
score(understanding, candidate) {
const raw = (
understanding.expandedTokens,
candidate,
deps.index.documentCount,
deps.index.averageLength,
deps.index.documentFrequency,
(text) => (text, deps.levers),
);
const max = (
understanding.expandedTokens,
deps.index.documentCount,
deps.index.documentFrequency,
);
const textFit = max > 0 ? raw / max : 0;
const freshness = (candidate.postedAt, snapshot, );
return {
item: candidate,
score: textFit + * freshness,
signals: (understanding, candidate, [
{ name: "bm25", value: textFit, weight: 1 },
{ name: "recency", value: freshness, weight: },
]),
};
},
};
}
a numbered note, explained on the righta function you can open heredefined further down in this filewhere level 2 joins