# Towards Green Text Generation: Memory-Based Speculative Decoding

**Authors:** Antal van den Bosch  
**Lab:** Hugging Face · **Published:** 12 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.16714) · [PDF](https://arxiv.org/pdf/2611.16714) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.16714.html)

## Key numbers

- **38.1%**: a domain-matched Olifant model reaches 38.1% acceptance on EU legal text
- **2.43×**: delivering 2.43x throughput and 2.51x lower energy per token than the neural verifier model alone
- **2.51×**: delivering 2.43x throughput and 2.51x lower energy per token than the neural verifier model alone

## Abstract

Speculative decoding accelerates text generation with large language models by letting a cheap draft model propose tokens that a large verifier checks in a single forward pass. We introduce Olifant, a draft model built on a memory-based classifier. It predicts the next token from a four-token window in under 0.3 ms on CPU, requiring no GPU, no backpropagation, and no accelerator memory. We measure throughput and energy under two verifier regimes on consumer hardware. Without a KV cache, drafting is nearly free: a domain-matched Olifant model reaches 38.1% acceptance on EU legal text, delivering 2.43x throughput and 2.51x lower energy per token than the neural verifier model alone. With a KV-cached verifier, itself a ~10x energy saver, drafting costs matter: a TinyLlama draft model halves throughput and triples energy despite high acceptance, whereas Olifant still yields 1.25x throughput and the lowest energy in the study (93 J per 100 tokens, 14x below the uncached baseline). We further show that KV caching decouples time from energy: verification of longer drafts raises power draw, shifting the energy-optimal draft length below the speed-optimal one. Speculative decoding is not dead in the age of fast verifiers, provided drafting is nearly free.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.