PostHogEvaluation
Do you even benchmark? How we evaluate agents at scale (with real examples)
Our short summary of PostHog’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on posthog.com (opens in a new tab)Problem
Agents were costly and their quality was hard to measure reliably.
Solution
PostHog built a benchmarking system with answer keys and multiple test methods to evaluate agent performance, cost, and speed on real code review tasks.
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on posthog.com (opens in a new tab)
Gusto