activeAgentsCoding

OpenWorldEngineer Mean@8

Metric: Mean@8 (%) (higher is better)Introduced: 2026

Mercor’s 26-task benchmark of long-horizon engineering and research projects. Each task’s raw score is rescaled against its baseline and ceiling; Mean@8 averages eight attempts per task, then tasks equally. Best@8 and Elo are separate metrics.

Models ranked

5

tracked on this benchmark

Score band

67.2 – 41.2

best → lowest tracked

Snapshot trend

—

need ≥2 snapshots

Leaderboard

Tracked models ranked by Mean@8 (%) (higher is better).

Compare candidates
#Model variant and provenanceScore
1

Configuration: GPT-6.2 Astra Max

Version: OpenWorldEngineer public 26-task Mean@8Harness: Official OpenWorldEngineer evaluation; 8 attempts per taskEvaluator: MercorObserved: Nov 18, 2026Confidence: confirmedSource

Notes: Official benchmark-owner results, independently evaluated. Mean@8 percentage, not Best@8 or Elo. Timestamp is retrieval date; page has no per-row run date. Reported 95% intervals are in the captured table.

67.2
2

Configuration: Fable 5.2 Max

Version: OpenWorldEngineer public 26-task Mean@8Harness: Official OpenWorldEngineer evaluation; 8 attempts per taskEvaluator: MercorObserved: Nov 18, 2026Confidence: confirmedSource

Notes: Official benchmark-owner results, independently evaluated. Mean@8 percentage, not Best@8 or Elo. Timestamp is retrieval date; page has no per-row run date. Reported 95% intervals are in the captured table.

54.6
3

Configuration: GPT-6 Astra Max

Version: OpenWorldEngineer public 26-task Mean@8Harness: Official OpenWorldEngineer evaluation; 8 attempts per taskEvaluator: MercorObserved: Nov 18, 2026Confidence: confirmedSource

Notes: Official benchmark-owner results, independently evaluated. Mean@8 percentage, not Best@8 or Elo. Timestamp is retrieval date; page has no per-row run date. Reported 95% intervals are in the captured table.

44.2
4

Configuration: Opus 5.5 Max

Version: OpenWorldEngineer public 26-task Mean@8Harness: Official OpenWorldEngineer evaluation; 8 attempts per taskEvaluator: MercorObserved: Nov 18, 2026Confidence: confirmedSource

Notes: Official benchmark-owner results, independently evaluated. Mean@8 percentage, not Best@8 or Elo. Timestamp is retrieval date; page has no per-row run date. Reported 95% intervals are in the captured table.

41.6
5

Configuration: GPT-6.1 Sol Max

Version: OpenWorldEngineer public 26-task Mean@8Harness: Official OpenWorldEngineer evaluation; 8 attempts per taskEvaluator: MercorObserved: Nov 18, 2026Confidence: confirmedSource

Notes: Official benchmark-owner results, independently evaluated. Mean@8 percentage, not Best@8 or Elo. Timestamp is retrieval date; page has no per-row run date. Reported 95% intervals are in the captured table.

41.2

How to read this benchmark

This benchmark scores models where higher is better. Use scores for directional filtering and shortlisting, not universal quality ranking; then validate pricing, context window, provider availability, and fit for your workload.

Trust this score when

  • There is a fresh timestamped snapshot (or multiple snapshots) for this benchmark.
  • The model list covers the same version family you can actually deploy today.
  • Top candidates overlap with your required routing and feature requirements.

Be cautious when

  • There is only one benchmark snapshot or the dataset appears stale.
  • The benchmark metric direction is opposite of your decision objective.
  • The score difference between options is narrow and likely within implementation variance.

Related benchmarks

Last reviewed: Nov 18, 2026

Resources