/ Methodology
How papers are ranked
Every candidate paper gets a score out of 100 from public numbers we look up each morning. An LLM does not pick the papers. It writes the summaries, and it lists the benchmark results an abstract claims, which code then checks word for word.
In short
- Do people use it? Code on GitHub, how fast it gains stars, and models or demos built on it.
- Are people talking about it? Upvotes on Hugging Face and points on Hacker News.
- Who wrote it? How often the authors' past work is cited, and whether a known lab is behind it.
- What does it show? Real benchmark results in the abstract, and how new the idea is compared with recent papers.
The details below are for readers who want the exact formula.
The formula
Four groups, each capped, so no single source decides the pick. Counts use a log scale: every doubling adds the same points, so one viral number can't swamp the rest.
| Signal | Max points |
|---|---|
| Builder evidence | 40 |
| Public code is linked from Hugging Face or the abstract | 8 |
| GitHub stars on that code, 2 points per doubling | 16 |
| Stars gained since the previous day, 1.5 per doubling (a repo under 2 weeks old counts all its stars) | 10 |
| Models 3, datasets 2, demos 1 on Hugging Face that link the paper | 6 |
| Community attention | 30 |
| Hugging Face upvotes, 3 per doubling | 20 |
| Hacker News points on stories linking the paper, 1.5 per doubling | 10 |
| Author track record | 15 |
| Median h-index of the authors on Semantic Scholar, 2 per doubling | 10 |
| A known lab 5, any named lab 2.5 | 5 |
| What the paper shows | 15 |
| 4 per benchmark result that is quoted word for word from the abstract | 10 |
| Where the paper stands among the day's candidates on distance from the last 90 days of papers (TF-IDF) | 5 |
If a lookup fails, or Semantic Scholar hasn't indexed a new paper yet, that signal gets the median of the day's papers and is marked estimated on the paper's page. An unknown value is never counted as zero. When the top papers are within 3 points, an AI editor compares them in pairs to break the tie, and the page says so.
Where the numbers come from
- GitHub: stars, stars gained since the day before, licence
- Hugging Face: upvotes, and models, datasets and demos that link the paper
- Hacker News: stories linking the paper: points and comments
- Semantic Scholar: authors' h-index now; citations for the 30-day scoreboard
- arXiv: titles and abstracts, for novelty and claims
Every day's numbers are saved in the open, in data/signals, so anyone can see how attention to a paper grew over its first days.
Does it work?
Once a week we check each saved day against what happened 30 days later, and compare the ranking with simpler ones on the same papers. The target is not Hugging Face upvotes, because the ranking reads those itself.
The first results arrive 30 days after the first full day is saved, because citations and stars need time to build up. The table appears here automatically once the weekly check has data.
Known limits
- These signals measure attention and adoption, not whether a paper is correct.
- Author h-index favours established researchers. It is capped at 10 of 100 points for that reason.
- Papers without code or a Hugging Face page have fewer ways to score, which favours applied work over theory.
- Novelty is measured on words, so a new idea described in familiar terms can look less novel than it is.
Get the data
Scores, signals and the scoreboard are all in the public repository. The code is in scorer.py.