Relevance Testing With Judgment Lists: A Starter Method

A judgment list is a fixed set of important queries, each paired with human ratings of which results are relevant. Testing search changes against it before release catches regressions that aggregate metrics miss, because the list encodes what your team believes good results look like for the searches that matter most.

Building a list worth trusting

Start with fifty to a hundred queries drawn from reality: your highest volume searches, your highest revenue searches, and a deliberate slice of hard cases, ambiguous terms, misspellings, and known problem queries from support. A list of only easy top queries will pass every change and teach you nothing.

For more on this, see this related guide and our earlier explainer on the same theme.

For each query, raters grade the top results on a simple scale: perfect, relevant, marginal, wrong. Use two raters on a sample to check they agree; where they disagree, discuss and write the reasoning into the list. That reasoning is the durable asset, because it captures merchandising judgment that otherwise lives in one person's head.

Scoring changes against the list

With a rated list, any ranking change can be scored automatically: run the new configuration over the fixed queries, compare the grades of the top results against the old configuration, and summarize with a metric like discounted cumulative gain or simple counts of improved and harmed queries. Harmed top revenue queries should block a release regardless of the average.

Keep the test honest by freezing the list during evaluation and versioning it when you deliberately update judgments. Quietly editing grades to make a change pass destroys the instrument. Changes to the list are decisions, reviewed like decisions, with dates and reasons attached.

Who judges, and how often

Judges should include whoever owns search outcomes: merchandising, search or product staff, and someone close to customers, like support leads who hear the complaints raw. Pure engineering judgment optimizes for clean logic; customer facing judgment remembers that the weird result is sometimes the one that sells.

Readers also found this companion piece from our archive useful.

Refresh a slice of the list quarterly: retire queries nobody searches anymore, add rising queries and last quarter's problem cases, and re rate results that merchandising strategy has redefined. A judgment list is a living contract about quality, and like personas and playbooks, it dies by neglect rather than by error.

What this catches that analytics cannot

Analytics tells you what happened across millions of searches, averaged into invisibility. The judgment list tells you that the new synonym expansion quietly broke your single most important brand query, on the day you ship it, before customers find it for you. Specific, named failures drive fixes; averages drive arguments.

It also aligns teams before releases. Debates about whether a change is better stop being taste contests when a shared, pre agreed standard scores both versions. Over time the list becomes the institutional memory of what good search means for your store, which outlasts any individual ranking tweak.

Frequently asked questions

How big should a judgment list be?

Start with fifty to one hundred queries: top volume, top revenue, plus deliberate hard cases like ambiguous terms and known problem searches. That size is ratable in days and sensitive enough to catch real regressions. Grow it quarterly as new important queries emerge.

Who should rate the results?

A mix: merchandising or search owners plus someone close to customers, such as support leads. Use two raters on a sample to check agreement, and record the reasoning behind disputed grades. The written reasoning becomes guidance that survives staff changes.

What metric scores a search change?

Compare graded results for old and new configurations on the same frozen queries, using discounted cumulative gain or counts of improved versus harmed queries. Protect top queries individually: a change that harms a top revenue query should not ship on a good average.

How often should the list be updated?

Review a slice quarterly. Retire dead queries, add rising and problem queries, and re rate where strategy has changed. Version every update with dates and reasons, and never edit grades mid evaluation to help a change pass; that destroys the test.