DeepSWE 1.1
DeepSWE 1.1 is Datacurve's June 2026 revised execution and grading setup for the same long-horizon DeepSWE tasks, grading committed patches in a clean isolated verifier environment and using mini-swe-agent for the public leaderboard. Model scores must preserve the mini-swe-agent harness/evaluator context.
Models ranked
32
tracked on this benchmark
Score band
79.0 – 42.2
best → lowest tracked
Snapshot trend
-6.05
Oct 23 → Nov 9 · 2 models
Leaderboard
Tracked models ranked by Pass@1 (higher is better).
Configuration: Muse Wonder Full (max)
Notes: Meta first-party benchmark table on the Muse Wonder model page; methodology in the Muse Wonder model card.
Configuration: Muse Wonder (max)
Notes: Meta first-party benchmark table on the Muse Wonder model page; methodology in the Muse Wonder model card.
Configuration: Gemini 4 Argon (highest thinking)
Notes: Blog benchmark table (gemini-4-argon_table_blog.gif); evals methodology page https://deepmind.google/models/evals-methodology/gemini-4-argon: self-computed with mini-swe-agent harness, highest thinking level. Retrieved 2026-10-17.
Notes: Moonshot self-reported launch table, row 'DeepSWE v1.1', column 'Kimi K3.1 Pro (max)'. Settings: reasoning effort max, temperature 1.0, top-p 0.95 (blog footnote). Moonshot-run with the Claude-Code harness, not the Datacurve mini-swe-agent board.
Configuration: claude-fable-5-2 (mean of 5 trials)
Notes: Anthropic self-reported (lab-published). System card, retrieved 2026-11-09. Does not move /best rankings (self-reported / raon rule). System card §8.3: DeepSWE v1.1, 113 long-horizon tasks, 74.3% averaged over five trials. Evaluator is Anthropic (lab run), not the public Datacurve/mini-swe-agent leaderboard. Do not treat as a board restamp.
Configuration: gpt-6-astra (Extra High)
Notes: Datacurve DeepSWE 1.1 leaderboard (113 tasks, mini-swe-agent, Pass@1 over 4 runs). Verified on https://deepswe.datacurve.ai/ (page generated_at 2026-09-22; page shows integer %%). Precise value from Epoch AI mirror deepswe_external.csv (Source column = deepswe.datacurve.ai). Highest-scoring published effort selected.
Configuration: gemini-3.8-flash (High)
Notes: Datacurve DeepSWE 1.1 leaderboard (113 tasks, mini-swe-agent, Pass@1 over 4 runs). Verified on https://deepswe.datacurve.ai/ (page generated_at 2026-09-22; page shows integer %%). Precise value from Epoch AI mirror deepswe_external.csv (Source column = deepswe.datacurve.ai). Highest-scoring published effort selected.
Configuration: MiniMax M3.1 Flash
Notes: MiniMax M3.1 Flash launch blog (2026-10-19) main benchmark table image m31-flash-en-2 (webp converted to png; values transcribed by eye from table) incl. footnotes. Footnote: mini-SWE-agent 2.4.5, max context 512K. Blog text also says 73.7%. Not on Datacurve DeepSWE board as of retrieval. Retrieved 2026-10-19.
Notes: Official GPT-5.6 GA launch benchmark row.
Configuration: Maximum effort
Notes: Vendor-reported. Source: pp.149-150, Table 8.1.A and Figure 8.2. The effort sweep ranges from 57.7 at low to 68.8 at maximum.
Configuration: kimi-k3 (Max)
Notes: Datacurve DeepSWE 1.1 leaderboard (113 tasks, mini-swe-agent, Pass@1 over 4 runs). Verified on https://deepswe.datacurve.ai/ (page generated_at 2026-09-22; page shows integer %%). Precise value from Epoch AI mirror deepswe_external.csv (Source column = deepswe.datacurve.ai). Highest-scoring published effort selected.
Notes: xAI launch post evals table reports Grok 4.6 High at 65.9% on DeepSWE v1.1. Variant: Grok 4.6 High (same model ID grok-4.6; high is the API default reasoning_effort).
Configuration: Nex-N2.5-Max
Notes: Nex-AGI HF model card benchmark table (column 3 = Nex-N2.5-Max). Sampling temperature 0.7, top_p 0.95, top_k 40 (note 2). Note 3: NexAU harness. Not the official mini-swe-agent setup; comparability caveat (10-17 Nex-N2.5-Pro precedent). Retrieved 2026-10-19.
Configuration: high thinking
Configuration: LongCat-2.5 (thinking on)
Notes: Self-reported by Meituan. Meituan LongCat first-party HF model card meituan-longcat/LongCat-2.5 (README sha 6fc4844c, lastModified 2026-11-09), benchmark table column 'LongCat-2.5'. Card default: enable_thinking=on, temperature 1.0, top_p 0.95 unless a footnote says otherwise. Footnote 2: 'DeepSWE v1.1: mini-swe-agent at temperature = 1.0, top_p = 0.95 and max_seq_length = 1M; results are averaged over three independent runs.' Row 'DeepSWE (v1.1)': 64.6. Not on Datacurve's board (deepswe.datacurve.ai leaderboard-live.json checked 2026-11-09). Retrieved 2026-11-09. Card caveat (verbatim): 'unmarked scores are measured in-house and are indicative only.' This LongCat-2.5 value is unmarked (no '*').
Notes: Meta HF model card Benchmarks table, xhigh reasoning effort.
Configuration: Qwen3.8-Flash
Notes: Qwen Team tech report 'Qwen3.8-Omni: Towards Native Omni-Modal Agents' (arXiv 2609.25611v1, 2026-09-22), Table 2 column 'Qwen3.8-Flash', row 'DeepSWE 1.1': 58.7. Footnote 1: highest of Claude Code and mini-SWE-agent runs; 'notably, Qwen3.8-Flash performs best on mini-SWE-agent', so this value is the mini-SWE-agent run. QwenCloud docs: 'DeepSWE 1.1 at 58.7'. Identity: the report's 'Qwen3.8-Flash' column and the QwenCloud qwen3.8-flash docs page carry the same values as the Qwen3.8-Flash-Next blog/HF card; Qwen's blog says Qwen3.8-Flash is the production served version (1M context, built-in tools) while these evals used a 256K context. Retrieved 2026-10-26.
How to read this benchmark
This benchmark scores models where higher is better. Use scores for directional filtering and shortlisting, not universal quality ranking; then validate pricing, context window, provider availability, and fit for your workload.
Trust this score when
- There is a fresh timestamped snapshot (or multiple snapshots) for this benchmark.
- The model list covers the same version family you can actually deploy today.
- Top candidates overlap with your required routing and feature requirements.
Be cautious when
- There is only one benchmark snapshot or the dataset appears stale.
- The benchmark metric direction is opposite of your decision objective.
- The score difference between options is narrow and likely within implementation variance.
Related benchmarks
Last reviewed: Jul 8, 2026