# LazyWeights: Decode-on-Demand Edge LLMs

**Authors:** Arnab Sanyal, Gourav Datta, Sandeep P. Chinchali  
**Published:** 12 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.17108) · [PDF](https://arxiv.org/pdf/2611.17108) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.17108.html)

## Key numbers

- **1%**: At 1% and 3.5% perplexity budgets
- **36%**: cuts Orin latency and energy per token by up to 36% and 38%
- **54%**: 54% and 44% with prompt-lookup speculation

## Abstract

Memory decides which LLMs an edge device runs, and how fast: capacity on a $4$ GB Jetson Nano, bandwidth on an $8$ GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, ttllama.cpp's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. ttLazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best ttllama.cpp format. A bit-exact $0.21$ mm^2 ASIC decoder unit ($1$ GHz, $7$ nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, ttLazyWeights needs 58-64% less time and 53-57% less energy per token than CPU ttllama.cpp, and runs a streamed Mistral-7B 6.6× faster at $3.89$ bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.