Cost and speed
Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
Our short summary of Apple’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on machinelearning.apple.com (opens in a new tab)Problem
Running large language models locally was too slow or required expensive cloud resources.
Solution
They used Apple’s MLX framework and RDMA over Thunderbolt to run trillion‑parameter models on one or multiple Mac Studios, achieving fast token generation.
36 tokens/second on 4 Mac Studios
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on machinelearning.apple.com (opens in a new tab)