Case studies / Evaluation
Evaluation
Offline evals, LLM judges, A/B tests: knowing a change is better before it ships.
Approach
32 write-ups
Tag:
-
LyftNewEvaluation
Beyond Test Cards: A Fake Payment Provider for Testing at Scale
Problem Payment tests could only succeed or fail completely, making it hard to verify retry and fallback logic.
Solution They created a fake payment provider that mimics any processor behavior, allowing realistic failure scenarios in production-like tests.
Read on eng.lyft.comDetails -
PinterestEvaluation
Scaling Model Evaluation at Pinterest with Ray (Part 4)
Problem Running many model experiments caused slow, costly evaluation pipelines.
Solution They combined Ray with Spark to compute metrics during inference, using accumulators to aggregate results on the fly, cutting evaluation time by up to 32%.
Evaluation time reduced up to 32%
Read on medium.comDetails -
MicrosoftNewEvaluation
From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence
Problem Need to test Copilot accuracy without exposing sensitive enterprise data.
Solution Created a synthetic data platform that generates realistic enterprise workspaces and runs automated evaluation pipelines to measure AI performance.
Read on microsoft.comDetails -
MetaEvaluation
Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale
Problem Code changes introduced bugs faster than traditional tests could catch, risking production issues.
Solution They built a system that uses LLMs to automatically write tests for each code change and an LLM‑as‑judge ensemble to filter out false positives.
Read on engineering.fb.comDetails -
PostHogEvaluationDo you even benchmark? How we evaluate agents at scale (with real examples)
Problem Agents were costly and their quality was hard to measure reliably.
Solution PostHog built a benchmarking system with answer keys and multiple test methods to evaluate agent performance, cost, and speed on real code review tasks.
Read on posthog.comDetails -
Grafana LabsEvaluation
Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1
Problem Teams couldn't tell if AI observability agents gave correct, reliable answers backed by telemetry.
Solution Grafana Labs created o11y-bench, an open benchmark that grades AI models on observability tasks with grounded, consistent, reproducible metrics, and added it to Fireworks' Specialized Intelligence Index.
Read on grafana.comDetails -
RazorpayEvaluation
How We Deploy Reliably at Razorpay
Problem Deployments often introduced bugs that broke payments, hurting merchants and customers.
Solution They built a progressive canary system that rolls out changes in tiny steps, uses AI judges to compare logs and metrics, and automatically rolls back if errors appear.
Read on engineering.razorpay.comDetails -
RazorpayEvaluation
Inside Razorpay’s RTO Model: The Engineering Behind its ML Lifecycle
Problem Showing cash‑on‑delivery could cause costly returns, but hiding it may lose sales.
Solution They built a risk model that scores each checkout, then a policy decides whether to show COD; outcomes are logged to evaluate and retrain the model.
Read on engineering.razorpay.comDetails -
ThumbtackEvaluation
Workbench: Building an AI Content Generation and Quality Assurance System at Scale
Problem Creating and reviewing hundreds of thousands of landing page descriptions manually was impossible.
Solution Thumbtack built Workbench, an AI pipeline that writes, automatically checks, and publishes marketing copy at scale, using automated quality checks and a bandit selector for variants.
97% drafts passed QA, $13K cost
Read on medium.comDetails -
GrabEvaluation
Powering AI-led research through simulation
Problem Testing new dispatch strategies took weeks, limiting how many ideas could be tried.
Solution Created a fast, offline simulator called sim‑rs that runs a full city‑day of orders in minutes, letting agents experiment and evaluate strategies quickly.
City‑day simulation completed in tens of minutes
Read on engineering.grab.comDetails -
Hugging FaceEvaluation
The Agent Said It Was Done. The Database Disagreed.
Problem AI agents seemed to work but often left incorrect data in the database.
Solution They built ThinkingBox, a benchmark that runs agents, checks the final database state and side effects, and scores consistency across many runs.
79,853 of 121,680 trials failed checks
Read on huggingface.coDetails -
Hugging FaceEvaluation
Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents
Problem Agents mix information from many tools, making it hard to tell if answers cite the correct source.
Solution They created ProvenanceGuard, a verification step that checks each claim against the specific tool output it claims to come from, catching wrong attributions.
Read on huggingface.coDetails -
NVIDIAEvaluation
How SWE-Serve Exposes the Gap Between Local Tests and Live Serving
Problem Coding agent patches passed unit tests but often failed when deployed to live model serving.
Solution Created SWE-Serve, a benchmark that runs full inference‑serving checks on patched code, testing model loading, decoding, caching, scheduling and API responses.
45.9% pass with full verifier
Read on developer.nvidia.comDetails -
NVIDIAEvaluation
How to Evaluate AI Agents From Tool Calls to Task Completion
Problem Teams couldn't tell if AI agents actually finished multi‑step tasks, not just gave plausible answers.
Solution They built a trace‑based benchmark that runs each tool call, tracks state, and scores both step‑level decisions and final task success.
Read on developer.nvidia.comDetails -
AmazonEvaluation
When LLM judges agree, should we believe them?
Problem Users needed a reliable way to judge if retrieved text was useful, but many LLM judges made similar mistakes.
Solution They built a system that checks how judges agree independently, then weights their votes based on that independence, improving accuracy by 9–14%.
Read on amazon.scienceDetails -
AmazonEvaluation
SOP-Bench: A new benchmark for evaluating AI agents on real business procedures
Problem AI agents struggled to follow real business procedures that need background knowledge and judgment.
Solution Amazon created SOP‑Bench, a benchmark with real SOPs, tools, and true answers, so agents are scored on actually completing tasks, not just generating text.
Read on amazon.scienceDetails -
AllegroEvaluation
Automating Search Relevance Assessment at Scale with LLM-as-a-Judge
Problem Search relevance was hard to measure accurately and cost‑effectively across many product categories.
Solution Allegro created a Relevance Assessment Tool that uses LLMs as judges to automatically score search results, applying detailed guidelines to handle diverse queries.
Read on blog.allegro.techDetails -
GrabEvaluation
Grab Bench: Evaluating AI on Grab-shaped production work
Problem Engineers couldn't trust AI outputs because subtle, plausible errors slipped past checks.
Solution They created Grab Bench, a configurable evaluation harness that runs AI models through task plugins, records results, and uses deterministic scorers or LLM judges to spot failures.
Read on engineering.grab.comDetails -
GustoEvaluationEval-Driven Design Systems (Part 2)
Problem Hard to know if AI‑generated UI code matches design rules and user intent.
Solution They built an evaluation pipeline that checks AI output against component schemas, runs static scorers, adds human ratings, and automatically captures real‑world prompts for continuous testing.
Read on engineering.gusto.comDetails -
GrabEvaluation
Crowdsourced taxonomy verification: A feedback-driven framework for refining knowledge graph relationships via online search interactions
Problem Wrong relationships in the knowledge graph gave users irrelevant search results and low trust.
Solution They created a loop that shows unverified graph links in search suggestions, watches user clicks, and uses that feedback to confirm or reject the links, continuously refining the graph.
Read on engineering.grab.comDetails -
AirbnbEvaluation
Eval-driven development: Lessons from evaluating GenAI at scale
Problem Airbnb needed a reliable way to test and improve unpredictable LLM features.
Solution They built an eval‑driven development framework that uses AI judges, synthetic tests, and human review to continuously catch and fix errors.
Read on medium.comDetails -
GustoEvaluation
Eval-Driven Design Systems (Part 1)
Problem AI coding assistants generate UI code that doesn’t follow the company’s private design system.
Solution Gusto added TypeScript schemas and automated evaluations to make the design system readable to AI and to check that generated code matches it.
Read on engineering.gusto.comDetails -
ScribdEvaluation
Stop Building Models. Start Building Systems.
Problem Need a scalable, cheap way to generate and validate labels as LLM models and prompts evolve.
Solution Created a repeatable human‑in‑the‑loop pipeline: fast model generates candidates, a higher‑capability judge model reviews samples, SMEs edit a slice, and results are ingested for iterative improvement.
Read on tech.scribd.comDetails -
Hugging FaceEvaluation
Featuring Every Eval Ever Results on Hugging Face Model Pages
Problem Evaluation scores were scattered in many formats, making them hard to compare and trust.
Solution Created a unified JSON schema and converters to collect, standardize, and display all eval results on model pages.
229,000 evaluation results stored
Read on huggingface.coDetails -
DropboxEvaluation
How we used DSPy to turn AI evaluations into better responses in Dash chat
Problem Chat agent often gave incomplete answers and used many tokens, hurting user experience.
Solution They built a layered evaluation system using LLM judges, calibrated it with human labels, then used DSPy to tune the agent’s prompts, creating a feedback loop that improved answer completeness and cut token use.
Fewer incomplete answers and reduced token usage
Read on dropbox.techDetails -
Booking.comEvaluation
The Causality Gap: Measuring the True Impact of Voluntary Adoption in Digital Marketplaces
Problem Users and partners chose whether to use new features, making it hard to see the real effect of those features.
Solution They ran a Randomized Encouragement Design and applied Double Machine Learning to separate the impact on people who adopted the feature from the overall rollout effect, revealing where to improve the product or the adoption funnel.
Read on booking.aiDetails -
ThumbtackEvaluation
Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI
Problem Generative AI could give inaccurate, biased, or unsafe answers, hurting trust in the marketplace.
Solution Thumbtack created a dedicated evaluation team with shared tools, curated datasets, and human oversight to continuously test AI safety, quality, and reliability.
Read on medium.comDetails -
LyftEvaluation
Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces
Problem Deciding how much to spend on driver incentives or rider prices without knowing long‑term effects was risky.
Solution They built a framework that splits regions and uses surrogate metrics to predict long‑term outcomes of pricing and incentive changes in the marketplace.
Read on eng.lyft.comDetails -
GoogleEvaluation
Improving breast cancer screening workflows with machine learning
Problem A shortage of radiologists threatens the double‑read breast cancer screening process.
Solution They built an AI system to act as a second reader, ran large retrospective and prospective studies, and showed it can match or improve detection while easing workload.
Read on research.googleDetails -
ZalandoEvaluation
Search Quality Assurance with AI as a Judge
Problem New markets lacked user data, so search quality was only checked after launch, causing slow fixes.
Solution They built an LLM-as-a-judge system that automatically evaluates search relevance at scale, using clustered real queries and NER to cover all search scenarios before launch.
Read on engineering.zalando.comDetails -
JioHotstarEvaluation
Gen AI Video — Building Scalable Validation Framework
Problem Generated videos often had subtle errors that broke realism, hurting viewer trust.
Solution They built a real‑time validation system that checks each video step for character consistency, scene flow, brand safety, and content appropriateness, and automatically flags or regenerates problematic parts.
Read on blog.hotstar.comDetails -
EtsyEvaluation
Reducing experiment duration with predicted control variates
Problem Experiments took too long, slowing product improvements.
Solution They added predicted control variates to their A/B testing platform, cutting metric variance and shortening experiment time.
Average experiment duration reduced by 3 days
Read on etsy.comDetails
No write-ups match these filters.