Evaluation
When LLM judges agree, should we believe them?
Our short summary of Amazon’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on amazon.science (opens in a new tab)Problem
Users needed a reliable way to judge if retrieved text was useful, but many LLM judges made similar mistakes.
Solution
They built a system that checks how judges agree independently, then weights their votes based on that independence, improving accuracy by 9–14%.
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on amazon.science (opens in a new tab)