Online Learning for Cost-Efficient LLM Routing

Our short summary of Ramp’s engineering write-up. The full story, with all the detail, is on their blog.

Problem

Requests were slow and expensive because the system chose suboptimal LLM models.

They built an LLM gateway that uses Thompson sampling on latency posteriors to route each request to the fastest, most reliable model.

Want the architecture, code and numbers? They’re in the original post.