LazyWeights: Decode-on-Demand Edge LLMs

Memory decides which LLMs an edge device runs, and how fast: capacity on a $4$ GB Jetson Nano, bandwidth on an $8$ GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, ttllama.cpp's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. ttLazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best ttllama.cpp format. A bit-exact $0.21$ mm^2 ASIC decoder unit ($1$ GHz, $7$ nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, ttLazyWeights needs 58-64% less time and 53-57% less energy per token than CPU ttllama.cpp, and runs a streamed Mistral-7B 6.6× faster at $3.89$ bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).