Cost and speed
Squeezing ~2× MFU out of our training runs
Our short summary of Nubank’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on building.nu.com (opens in a new tab)Problem
Training large Transformer models was slow and under‑utilized the hardware.
Solution
They rewrote the attention kernel, integrated it with the compiler, and tuned the optimizer, which doubled the model‑flops‑utilization during training.
~2× MFU improvement
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on building.nu.com (opens in a new tab)