Cost and speed
Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost
Our short summary of Shopify’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on shopify.engineering (opens in a new tab)Problem
Long prompts made the agent slow and expensive to run.
Solution
They replaced thousands of prompt tokens with a few learned “gist” tokens, training the embeddings by distilling from the full prompt so the model behaves the same.
Latency dropped from 6.8s to 4.2s, throughput rose from 20.2 to 23.4 QPS
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on shopify.engineering (opens in a new tab)