HumanEval+
About
HumanEval+ extends HumanEval with an average of 764 test cases per problem (vs 9.6 in original), greatly increasing evaluation rigor for edge case coverage.Compare model scores alongside pricing, API availability, and release date — high benchmark score alone doesn't make a model the right pick.
How to read this benchmark
This benchmark uses a scoring system where higher is better. Scores are useful for directional filtering and model shortlisting, not for universal quality ranking.
Interpretation checklist: prefer benchmarks that are closest to your workload style, then validate against the linked model pages for pricing, context window, and provider availability.
Trust when:
- There is a fresh timestamped snapshot (or multiple snapshots) for this benchmark.
- Model list covers the same version family you can actually deploy today.
- Top candidates overlap with your required routing and feature requirements.
Don't trust when:
- There is only one benchmark snapshot or the dataset appears stale.
- Benchmark metric direction is opposite of your decision objective.
- The score difference between options is narrow and likely within implementation variance.
Current modeled score band for tracked entries is roughly 87.2 – 87.2.
Window
May 28
single timestamp
Mean delta
No trend
need another snapshot
Coverage
2
models in latest snapshot
Leaderboard preview (top 5)
- 1. GPT-4o (05-13)87.2
- 2. Qwen2.5-Coder-32B-Instruct87.2