Methodology

Every page on Databubble — the bubble chart, the rankings rail, the model detail card — is ordered by a single number we call databubble_score. It is a 0-100 composite, recomputed across the entire model universe once a day, that reflects how a given AI model is doing along six independent dimensions at once.

We built it because every single-metric leaderboard misleads. Sort by downloads and the top of the list is tiny embedding models from RAG tutorials. Sort by trending and you see whatever was released yesterday. Sort by MMLU and the front page fills with leaderboard-hunting fine-tunes nobody uses in production. Each ordering is technically correct and substantively misleading, because no single signal fully captures what people mean when they ask "is this model a big deal right now?"

The composite is opinionated. We chose six signals, fixed weights, and a uniform normalization. None of these choices are neutral, and the rest of this page exists to make them legible. We do not run our own benchmarks; we aggregate signals published by HuggingFace, LMSYS, the Open LLM Leaderboard maintainers, the SWE-bench team, Semantic Scholar and several pricing-and-throughput providers, and link back to each source so any number can be verified without trusting our pipeline.

If you only read one section, read Limitations & trade-offs — a composite hides almost as much as it reveals, and we want users to know what the score does not say.

Each signal answers a different question. Together they cover novelty, adoption, human preference, academic capability, agentic skill and confidence-of-measurement. For each: what it measures, source, normalization, why this weight, and what it correlates with.

The score is a weighted average of min-max-normalized signals. For a single model m, with the set of signals where the value is non-null written as S(m), the formula is:

databubble_score(m) =
  ( Σ over s ∈ S(m) of  weight[s] × normalized[s] )
  ─────────────────────────────────────────────────
  ( Σ over s ∈ S(m) of  weight[s] )

where  normalized[s] = ((value[s] − min[s]) / (max[s] − min[s])) × 100
       min[s], max[s] are computed across the entire active model set
       weight[s] is the signal's published percentage (25, 25, 20, 15, 10, 5)

Two implementation details matter. First, min and max are recomputed every run from the live model set, not held as historical constants. A new flagship that pushes the maximum Elo from 1450 to 1500 implicitly rescales every other model's normalized Elo downward — a feature, since the score is meant to be relative to the current frontier.

Second, weights are redistributed when a signal is missing. A model with no SWE-bench entry does not get a 0; instead, the 10% weight drops out of both numerator and denominator. This prevents partial-coverage bias against models not evaluated on every available benchmark.

Edge case: if every signal is null, or every available signal has min == max (zero variance), the score is reported as 0 rather than a divide-by-zero error. In practice this only happens for stub records during ingestion catch-up.

The implementation is small enough to be read in one sitting: frontend/src/app/api/ingest/compute-trends/route.ts. The function computeRanges computes the per-signal min/max in a single pass; computeDatabubbleScore applies the weighted formula above and rounds the result to one decimal place.

To make the formula concrete, here is a hypothetical model. All numbers are illustrative round figures, not the actual values for any real release. Read this as "what if a frontier-class release scored roughly here."

Hypothetical model: a chat-oriented LLM, a few weeks post-release, evaluated on every public leaderboard.

Raw values (example, not real data)
  trending_score      = 480       (top quartile but past the launch spike)
  arena_elo           = 1320      (strong, well below the absolute frontier)
  avg_benchmark       = 60        (open-llm-leaderboard composite, 0-100 scale)
  day_change          = 8         (8% more downloads vs 24h ago)
  swe_bench_resolved  = 35        (35% issues solved on the best of 6 boards)
  arena_votes         = 18000

Active universe ranges (example)
  trending_score      [0, 1500]
  arena_elo           [900, 1450]
  avg_benchmark       [0, 80]
  day_change          [-50, 200]
  swe_bench_resolved  [0, 75]
  arena_votes         [0, 80000]

Normalized signals (0-100)
  trending_score      = (480 − 0) / (1500 − 0)  × 100 = 32.0
  arena_elo           = (1320 − 900) / (1450 − 900) × 100 = 76.4
  avg_benchmark       = (60 − 0) / (80 − 0)   × 100 = 75.0
  day_change          = (8 − (−50)) / (200 − (−50)) × 100 = 23.2
  swe_bench_resolved  = (35 − 0) / (75 − 0)   × 100 = 46.7
  arena_votes         = (18000 − 0) / (80000 − 0) × 100 = 22.5

Weighted sum
  numerator   = 25 × 32.0 + 25 × 76.4 + 20 × 75.0 + 15 × 23.2 + 10 × 46.7 + 5 × 22.5
              = 800 + 1910 + 1500 + 348 + 467 + 112.5
              = 5,137.5
  denominator = 25 + 25 + 20 + 15 + 10 + 5 = 100

databubble_score = 5,137.5 / 100 = 51.4

Two things to notice. First, arena_elo and avg_benchmark do most of the heavy lifting — together they account for roughly (1910 + 1500) / 5137.5 ≈ 66% of the final score. That is by design. Second, despite a strong Elo, the score lands in the mid-50s because trending and adoption signals are middle of the pack. A score in the 80s is reserved for models that are simultaneously high-quality and being actively adopted.

If this hypothetical model had no SWE-bench entry, the 10% weight would drop out and the denominator would be 90. The new score would be (5,137.5 − 467) / 90 ≈ 51.9. Almost identical — which is what weight redistribution is supposed to deliver.

Every signal is on its own ingestion cron, scheduled inside the production container via Coolify. The schedules below are the actual production cadence as of this writing. Source code lives under /api/ingest/*; the schedule itself is in frontend/docs/cron-schedule.md.

The composite itself is recomputed once a day, at 1 AM UTC, after the daily snapshot is written. That means a freshly-ingested benchmark score at 2 PM UTC will not affect databubble_score until the next overnight recompute — by design, so the score does not jitter intraday.

No score this compact is right for everyone. The composite is meant to surface "models worth paying attention to right now," not to settle benchmark debates. Here are the trade-offs we have made and the ways the score can mislead.

Trending vs. quality, again. Even with quality signals at 45% (25% Elo + 20% benchmark), a viral release can briefly score higher than a more capable but quieter model. The score is not "best model"; it is "model that is a big deal right now," and what people are using is a legitimate part of that.

English-language Arena bias. LMSYS Chatbot Arena is overwhelmingly English-language general-chat. Models excellent at non-English, code-only or long-form structured tasks can be underrated by Elo. The benchmark composite (with MUSR and BBH) and swe_bench_resolved partially compensate, but for niche use-cases the underlying signals are more informative than the composite.

Benchmark gaming. The Open LLM Leaderboard composite is robust by construction (six contamination-resistant tasks), but no benchmark is uncheatable. Fine-tuning on a leaked answer key or training on the public dev split produces inflated scores. The 25% Arena weight is a partial defense: a model that aces benchmarks but loses pairwise comparisons rapidly falls out of the top.

Missing-data redistribution can flatter sparse models. Redistribution stops partial-coverage models from being zeroed-out, but a model with only one strong signal (say, a high trending_score and nothing else) can read higher than is warranted. We require multiple non-null signals before a model appears in headline rankings, but it is worth noting when comparing a brand-new release to an established one.

No multimodal coverage in the score. Image, audio and video models live on Databubble, but none of the six composite inputs measure multimodal quality directly. Their scores end up driven primarily by trending_score and day_change, which is honest about the data we have and dishonest about the actual quality of the model. This is the single biggest known gap.

Proprietary models lack one signal universally. Closed models (e.g. proprietary commercial APIs) have no HuggingFace trending_score, so for them the 25% weight always redistributes. Their composite is anchored more strongly on Arena Elo and benchmarks. We think this is the right call — a model with no public Hub presence cannot have a Hub trending value — but it is a structural asymmetry users should know about.

Min-max normalization is sensitive to outliers. A single anomaly at the top can compress the rest of the distribution. We accept this because rank- or percentile-based alternatives make the score unstable as the universe size changes, which it does daily.

The composite is versioned (databubble_score v1, as of this writing) and we expect to revise it. Concrete signals on the candidate list, in rough order of priority:

Multimodal evaluation. A vision-and-language counterpart to swe_bench_resolved, drawn from public multimodal leaderboards (MMMU, MathVista, MMVet, equivalents for audio and video). Without this, the composite genuinely under-serves image/audio/video models.

Agentic / tool-use evaluation. Beyond SWE-bench, results from agent benchmarks like GAIA, AgentBench and MLE-bench. Reasoning-only and tool-augmented capability are increasingly divergent and worth measuring separately.

Long-context evaluation. Standardized "needle in a haystack" and long-form-reasoning scores at 100k+, 1M+ context. The current composite barely registers context capability, which is becoming a primary axis of differentiation.

Cost-adjusted quality. We already store pricing (price_input, price_output) and throughput (speed_tps, latency_ms). An Elo-per-dollar or benchmark-points-per-token signal would reflect a real-world decision criterion the composite ignores.

Citation velocity. We ingest citation_count and influential_citations from Semantic Scholar but do not yet use them in the score. Adding a citation-velocity signal would weight academic influence alongside community adoption.

Smarter normalization. Replacing min-max with a percentile mapping (cap at p99, floor at p1) would reduce outlier sensitivity without adding category-specific buckets.

When the formula changes meaningfully we will bump the version on this page and call out the diff. The point of a methodology page is for the reader to disagree with specific, dated artifacts.