All papersLLMs & Language Models

Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them

/ Paper arXiv 2609.05881

Key numbers

2.3%Seven official tags (2.3%)

From the paperCopied from the abstract, not written by AI.

Abstract

From the paperThe authors' own words.

Developers download quantized large language models (LLMs) from public registries to run them locally. None of the registries we examined documented functional testing of their files. We tested all 305 eligible code-capable GGUF tags (the llama.cpp format) across 15 model lines in the official Ollama library, plus 22 popular Hugging Face community files. A 15-task code screen identified suspects, which we retested on all 164 HumanEval+ tasks. Seven official tags (2.3%) solved at most six of them. All were low-bit quantizations of models with at most four billion parameters. We rebuilt five suspect files from the original weights. All five registry files matched our uncalibrated rebuilds tensor for tensor, so none was corrupted. Exact reproducibility did not imply function. With the weights, toolchain and tensor mix held fixed, adding an importance matrix raised three Qwen2.5-Coder-3B builds from zero to 76-113 solved tasks. The same uncalibrated recipe left the smaller Qwen2.5-Coder-1.5B functional, showing that the recipe alone does not predict failure. None of 66 sampled tags that passed the screen was non-functional. We release the audit data and the quantcheck tool and recommend screening low-bit tags before publication, then calibrating or withholding failures.

From the full paper

From the paperCopied from the paper itself by our code.

Reading the paper on arXiv…

Try it

Code, models and demos that use this paper.

Datasets:
aditiPatodiya/quantcheck-audit ↗1 dataset on Hugging Face

Ask this paper

/ Ask a questionAnswers come from the full paper and name the section

    / Your notes

    Select text in the abstract to highlight it or add a note. They stay in this browser.

      Read the PDF