💰 SubDollarBench sdb-sql campaign careers
Budget LLMs build a SQL database, ticket by ticket, for at most $0.99 per phase. Graded against SQLite.
📈 Results
Updated 2026-11-09 09:21 UTC
Top model
deepseek/deepseek-v4.1-flash-1017
65.0 / 76 median
Best median points per dollar
deepseek/deepseek-v4.1-flash-1017
29.3 pts/$
Cheapest career
openai/gpt-6-luna
$0.61
Careers
18 across 6 models
$42.83 spent
| # | Model | Points (of 76) | Perf | Spend | Tickets agreed | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
#1 | deepseek/ highest ticket 20kept 19 green32.2 pts/$4.9 h1,816 turnsout 1.6M · in 4.3M (+141.1M cached)done 63 - 70 · 3 careers | 65.0 / 76 | 1.08× | $2.02 | agree 18/20 · differs: 15, 19 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
×3 careers ▸01234567891011121314151617181920 agree 18/20 · differs: 15, 19 perf 1.08× $ $2.02 duration 4.9 h turns 1,816 tokens out 1.6M
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
#2 | xiaomi/ highest ticket 20kept 18 green21.4 pts/$16.3 h1,269 turnsout 2.9M · in 12.8M (+126.2M cached)done 24.4 - 67 · 3 careers | 62.8 / 76 | 0.10× | $2.94 | agree 6/20 · differs: 1, 2, 3, 4, 5, 6, 7, 10, 12, 14, 17, 18, 19, 20 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
×3 careers ▸01234567891011121314151617181920 agree 6/20 · differs: 1, 2, 3, 4, 5, 6, 7, 10, 12, 14, 17, 18, 19, 20 perf 0.10× $ $2.94 duration 16.3 h turns 1,269 tokens out 2.9M
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
#3 | openai/ highest ticket 20kept 18 green22.9 pts/$8.2 h1,012 turnsout 1.9M · in 4.3M (+112.7M cached)done 55.8 - 62.2 · 3 careers | 58.4 / 76 | 0.13× | $2.55 | agree 16/20 · differs: 10, 14, 18, 19 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
×3 careers ▸01234567891011121314151617181920 agree 16/20 · differs: 10, 14, 18, 19 perf 0.13× $ $2.55 duration 8.2 h turns 1,012 tokens out 1.9M
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
#4 | anthropic/ highest ticket 19kept 19 green23.3 pts/$6.3 h650 turnsout 3.9M · in 1.8M (+31.9M cached)done 55.6 - 64.6 · 3 careers | 57.6 / 76 | 0.16× | $2.47 | agree 18/20 · differs: 17, 20 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
×3 careers ▸01234567891011121314151617181920 agree 18/20 · differs: 17, 20 perf 0.16× $ $2.47 duration 6.3 h turns 650 tokens out 3.9M
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
#5 | openai/ highest ticket 15kept 16 green27.5 pts/$5.5 h1,524 turnsout 848K · in 2.3M (+94M cached)done 27.4 - 47 · 3 careers | 44.0 / 76 | 0.35× | $1.60 | agree 13/20 · differs: 2, 5, 6, 10, 12, 14, 17 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
×3 careers ▸01234567891011121314151617181920 agree 13/20 · differs: 2, 5, 6, 10, 12, 14, 17 perf 0.35× $ $1.60 duration 5.5 h turns 1,524 tokens out 848K
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
#6 | qwen/ highest ticket 19kept 13 green12.0 pts/$8.0 h1,309 turnsout 1.5M · in 2.7M (+124.7M cached)done 25 - 37 · 3 careers | 37.0 / 76 | 0.15× | $3.09 | agree 8/20 · differs: 1, 2, 3, 5, 6, 7, 8, 11, 12, 14, 17, 19 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
×3 careers ▸01234567891011121314151617181920 agree 8/20 · differs: 1, 2, 3, 5, 6, 7, 8, 11, 12, 14, 17, 19 perf 0.15× $ $3.09 duration 8.0 h turns 1,309 tokens out 1.5M
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ppasslliteffailxfaultsskippedrrunningnnot reachedtN p = ticket N pass · tN k/m = k of m careers passed
5 $0 rehearsals (the reference server; 71 / 76 each) are not ranked.
How this board is ranked
- Careers: a career is 4 phases and 21 tickets (0-20); one row per leaf phase cell of a chain. A model's row is its median career by the one shared board order; the best and worst careers are named next to it and the full set sits in ×N careers ▸.
- Points: difficulty-weighted score. On equal points the Perf score decides (higher first; a career with no score ranks below one with a score), then Pts / $ divides by effective spend. The perf contract is ticket 0's Performance section; hover a Perf badge for each workload's reference, candidate, ratio and any floor.
- Rank:
#1with a70%badge means the model holds rank 1 in 70% of all combinations of one career per model, ranked by the board order (points, then perf, then points per dollar); hover the badge for the full distribution.=marks two models that most likely share a rank. - Only the current rule set is shown. A career still running is counted in the in-progress banner, never shown as a row; a $0 rehearsal is a footnote.
- Hidden, poisoned and harness-fault cells are never published. The ladder squares merge each phase cell's own tickets; the legend sits under the table.