Search: tokens, ranking and index lag

The index maps each term to postings: document IDs, often with term frequencies and positions. For an AND query, intersect term lists. For OR, take their union. This avoids reading every document for each query.

Tokenisation splits text; filters may lowercase, stem or remove stop words. This analyzer does not stem: bikes and bike remain different terms. Query analysis must be compatible with indexing analysis. Changing an index-time analyzer usually requires rebuilding indexed terms.

TF-IDF combines term occurrence with corpus rarity. BM25 adds a saturating frequency contribution and document-length normalization. Scores estimate lexical relevance, not truth or business value. Repeating one word endlessly should not produce an unbounded BM25 advantage.

Search often reads a derived index. Delivery delay and refresh delay both contribute to visibility lag. A successful database write does not imply immediate search visibility. Serve authoritative post-write confirmation from the source, or wait for a supported visibility acknowledgment.

A coordinator queries relevant shards and merges candidates. Each shard can search a replica. More shards add coordination and tail-latency exposure; they do not guarantee faster requests. With comparable scores, exact local top-k results suffice for global top-k. Approximate retrieval or inconsistent statistics can change ranking.

Choose a product policy for shard failures. Partial responses can be useful, but must disclose incompleteness. Source deletions and access-control changes also need careful propagation: a delayed index must not become the authority for permissions.

Try a fresh variant, then score at least 80% on the mastery check. Wrong answers link back to the teaching step.