NL2Repo-Bench
Natural-language-to-repository benchmark measuring agentic ability to scaffold and implement full repositories from specifications. DeepSeek reports scores with DeepSeek Harness Minimal mode and a 1M-token context window.
Models ranked
3
tracked on this benchmark
Score band
68.2 – 61.2
best → lowest tracked
Snapshot trend
-5.60
Oct 17 → Oct 19 · 1 models
Leaderboard
Tracked models ranked by Score (higher is better).
Notes: Value per DeepSeek V4.1 tech report Table 3 and DeepSeek changelog 2026-09-10 (65.4). The base Hugging Face README table lists 64.0 for the same cell; DeepSeek has not corrected either. Checked 2026-10-17.
Configuration: MiniMax M3.1 Flash
Notes: MiniMax M3.1 Flash launch blog (2026-10-19) main benchmark table image m31-flash-en-2 (webp converted to png; values transcribed by eye from table) incl. footnotes. Footnote: reward-hacking cases assigned reward 0; vendor corrected two tasks with spec/test inconsistencies and scored on the corrected set (comparability caveat). Retrieved 2026-10-19.
How to read this benchmark
This benchmark scores models where higher is better. Use scores for directional filtering and shortlisting, not universal quality ranking; then validate pricing, context window, provider availability, and fit for your workload.
Trust this score when
- There is a fresh timestamped snapshot (or multiple snapshots) for this benchmark.
- The model list covers the same version family you can actually deploy today.
- Top candidates overlap with your required routing and feature requirements.
Be cautious when
- There is only one benchmark snapshot or the dataset appears stale.
- The benchmark metric direction is opposite of your decision objective.
- The score difference between options is narrow and likely within implementation variance.
Related benchmarks
Last reviewed: Sep 10, 2026