Guardrails
Teaching Sidekick to say no: automated data curation with LLM judge consensus
Our short summary of Shopify’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on shopify.engineering (opens in a new tab)Problem
The assistant never learned to refuse impossible merchant requests, leading to bad answers.
Solution
Created an automated pipeline where several LLMs vote on whether a query should be refused, generated refusal examples, and fine‑tuned the model with this data.
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on shopify.engineering (opens in a new tab)