Effectiveness leaderboard
Which skill sets actually work, per project type — measured in our own harness, not marketed. This table compares routing strategies (arms), not individual skills: per-skill top-tier ranking needs fleet telemetry (usage + retention), which ships in a later phase behind the flag below.
About this data
- Source
test/harness/runs/series3.jsonl- What it is
- Series 3 — internal harness eval (pilot scale, n=3 per cell). Not fleet telemetry.
- Policy
- Delayed and aggregated. No raw runs, no user data — see publishing strategy (FAZ 9 plan).
- Generated
- 2026-10-04
Pilot scale: 3 runs per cell — each cell shows the mean and the observed min–max spread across the 3 runs, not a confidence interval; rankings at this n are noise, not verdicts. The none arm is a negative control (no skill cards shown), so its routing recall is 0 by construction, not a finding. Arms compare routing strategies, never individual catalog skills.
Why only Series 3? Series 3 is shown because it runs the final accepted methodology (4 arms incl. none negative control, ground-truth reachability fix). Series 1 and 2 are earlier pilots — different scenarios, pre-fix methodology — retained locally at test/harness/runs/series1.jsonl and series2.jsonl for audit (git-ignored, not in the repo), not cherry-picked out of this page.
docs
| Arm | Routing recall min–max | Task score min–max | Token cost mean est. | Runs | Notes |
|---|---|---|---|---|---|
jev |
1.00 [1.00–1.00] | 0.53 [0.40–0.60] | 1,613 tok | 3 | — |
naive |
0.33 [0.00–1.00] | 0.40 [0.40–0.40] | 1,833 tok | 3 | — |
none |
0.00 [0.00–0.00] | 0.33 [0.20–0.40] | 1,240 tok | 3 | structural zero (negative control: no skill cards shown) |
repotify |
1.00 [1.00–1.00] | 0.60 [0.60–0.60] | 1,932 tok | 3 | Repotify-recommended set |
python-api
| Arm | Routing recall min–max | Task score min–max | Token cost mean est. | Runs | Notes |
|---|---|---|---|---|---|
jev |
1.00 [1.00–1.00] | 0.76 [0.71–0.86] | 3,090 tok | 3 | — |
naive |
0.00 [0.00–0.00] | 0.48 [0.29–0.71] | 1,691 tok | 3 | — |
none |
0.00 [0.00–0.00] | 0.29 [0.29–0.29] | 971 tok | 3 | structural zero (negative control: no skill cards shown) |
repotify |
1.00 [1.00–1.00] | 0.57 [0.43–0.71] | 3,469 tok | 3 | Repotify-recommended set |
testing
| Arm | Routing recall min–max | Task score min–max | Token cost mean est. | Runs | Notes |
|---|---|---|---|---|---|
jev |
0.67 [0.00–1.00] | 0.72 [0.67–0.83] | 1,936 tok | 3 | — |
naive |
0.00 [0.00–0.00] | 0.83 [0.67–1.00] | 1,699 tok | 3 | — |
none |
0.00 [0.00–0.00] | 0.89 [0.83–1.00] | 1,018 tok | 3 | structural zero (negative control: no skill cards shown) |
repotify |
1.00 [1.00–1.00] | 0.72 [0.67–0.83] | 2,246 tok | 3 | Repotify-recommended set |
How we measure
No stars, no ratings. Every number on this page is measured — this is the machinery behind it.
1 · Rule scan, then a 3-model jury
Every catalog entry first passes a rule-based security scan (verified / caution / quarantined / rejected). A jury of two or three models then scores quality, specificity and maintenance as 0–1 medians — the mean is the score you see on skill pages. Agreement (how much the models concur) gates the score: below 0.6 it discounts quality. Scores are valid for 30 days; skill pages mark an expired score until the catalog is rebuilt.
2 · The coverage gate — measured in a pilot
Recommendation sets are context-budgeted, so a coverage gate decides which skills stay. Four gate variants ran in a pilot (30 runs per arm, synthetic take-all mock agent) on must-have recall:
| Gate variant | Recall | Precision | F1 | nDCG | Avg. set size |
|---|---|---|---|---|---|
strict | 0.500 | 0.129 | 0.194 | 0.214 | 7.7 |
loose | 0.786 | 0.098 | 0.169 | 0.214 | 12.0 |
jaccard (default) | 1.000 | 0.115 | 0.201 | 0.270 | 12.4 |
hybrid | 0.857 | 0.082 | 0.146 | 0.214 | 14.7 |
Jaccard reached recall 1.000 after a targeted triage fix (tie-contender escape) that repaired a real miss — react-best-practices dropped for a look-alike skill — with no scenario regressing. Hybrid was eliminated: bigger sets crowded must-have items out of the context budget. A separate real-world check (4 realistic projects x 4 variants, judged by a 3-model jury: nemotron-3-super-120b-a12b, gemma-4-31b-it, gpt-oss-20b) agrees — average jury coverage: jaccard 7.0 > loose 6.6 > hybrid 6.4 > strict 5.5. Decision: jaccard is the default — an item whose max pairwise Jaccard overlap with an already-selected skill reaches 0.6 is dropped. Reversible at any time via the REPOTIFY_COVERAGE_VARIANT environment variable.
3 · Every gate decision leaves an audit trail
Drops are never silent. Each gate decision carries a reason code — JACCARD_PASS, NO_NEW_COVERAGE_JACCARD, STRICT_PASS, NO_NEW_COVERAGE_STRICT, SCORE_BAND_PASS, CORE_TIER_PASS, BUDGET_EXCEEDED — and Jaccard drops additionally record the measured overlap value and the blocking skill's id. The decisions are logged with each recommendation on the user's machine, so any set can be re-examined there. No fake precision: the gate's own calibration debt (the 0.6 threshold sits on a narrow working window) is tracked openly in the repo, not hidden.
4 · Per-skill rankings wait for fleet data
Ranking individual skills needs fleet telemetry — installs, usage, retention — aggregated and delayed. Until that data crosses the threshold below, the fleet section stays parked: we show no per-skill rankings rather than invented ones.
Fleet leaderboard flag: off
Usage + retention across real installs — effectiveness, not popularity. Unlocks when fleet telemetry crosses the data threshold (200 installs, FAZ 9). Until then this section stays parked: no fake numbers.