Evaluation
Automating Search Relevance Assessment at Scale with LLM-as-a-Judge
Our short summary of Allegro’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on blog.allegro.tech (opens in a new tab)Problem
Search relevance was hard to measure accurately and cost‑effectively across many product categories.
Solution
Allegro created a Relevance Assessment Tool that uses LLMs as judges to automatically score search results, applying detailed guidelines to handle diverse queries.
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on blog.allegro.tech (opens in a new tab)
PostHog
Gusto