Benchmarks

Long-context memory benchmark for conversational agents evaluating ability to recall and reason over extended multi-session dialogue histories.

Real-world dialogue evaluation benchmark for conversational agents. Limited public documentation available.

UC Berkeley agentic decision-making reliability and safety benchmark testing coherence, persistence, and safety in long-horizon tasks. NOTE: slug contains parentheses — recommend renaming to 'adrs'.

Crowdsourced human preference leaderboard using pairwise blind comparisons of LLM responses. Operated by LMSYS/UC Berkeley; rebranded and moved to lmarena.ai in 2024.

80 multi-turn conversation questions across 8 categories evaluated by GPT-4 as judge. Scores range from 1-10 per turn.

164 Python coding problems measuring functional correctness of code generation via pass@k metric. Released by OpenAI in 2021; HumanEval+ provides a more rigorous extension.

974 Python programming problems for evaluating code generation, ranging from beginner to intermediate difficulty.

HumanEval+ extends HumanEval with an average of 764 test cases per problem (vs 9.6 in original), greatly increasing evaluation rigor for edge case coverage.

MBPP+ extends MBPP with 21.2 average test cases per problem, a 35x increase over the original for more rigorous code generation evaluation.

Large-scale text-to-SQL benchmark with 10,181 questions across 200 databases from 138 domains, requiring cross-domain generalization.

Conversational tool-use benchmark with 78 conversations requiring multi-step API calls across 28 tools including calendar, email, and messaging.

Continuously updated coding benchmark sourcing new problems from LeetCode, AtCoder, and Codeforces post-May 2023 to eliminate contamination. Evaluates code generation, self-repair, and execution prediction.

800 Python function input-output reasoning problems testing code execution understanding rather than code generation. Two tasks: input prediction and output prediction.

500 human-validated GitHub issue resolution tasks from SWE-bench, created with OpenAI in August 2024. The standard evaluation for agentic coding systems. Top performers (2026) exceed 78% resolved.

Real-world code editing benchmark measuring a model's ability to apply changes to existing codebases across 8 programming languages using Exercism platform exercises.

1,140 complex Python programming tasks spanning diverse real-world domains requiring multi-library function calls. Two variants: Complete (function completion) and Instruct (natural language to code).

731-task multilingual real-world GitHub issue benchmark extending SWE-bench Verified with harder, more diverse tasks across Python, JavaScript, TypeScript, Java, Go, C++, and Rust.

23 challenging tasks from BIG-Bench where even large models scored below human performance without chain-of-thought. Tests logical deduction, causal judgment, and formal fallacies.

Automated LLM evaluation using 1,024 real user queries from WildChat with LLM-as-judge scoring. Achieves 0.98 Spearman correlation with Chatbot Arena Elo. V2 with periodic updates.

Evaluation suite based on 20 official human-centric standardized exams (SAT, LSAT, GRE, bar exam) spanning English and Chinese.

Tests LLMs on undergraduate to professional level knowledge across 57 subjects with 15,908 multiple-choice questions. Top models now saturate at 86–90%; MMLU-Pro is the recommended harder successor.

IFEval measures LLM ability to follow verifiable formatting instructions (e.g., output length, keyword inclusion, capitalization) with exact-match checking.

Enhanced MMLU with 10-option questions (vs 4 in original), harder problems filtered through expert review, and reduced shortcuts. Maintains coverage of 57 academic subjects.

Stanford framework evaluating LLMs across 30+ scenarios spanning 7 dimensions: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. NOTE: slug contains parentheses — recommend renaming to 'helm'.

95,000 trivia question-answer pairs with associated evidence documents from Wikipedia and the web for reading comprehension evaluation.

ZeroSCROLLS suite's QuALITY subset: multiple-choice QA on long-form documents (articles, books) requiring reading up to 10K tokens for answer derivation.

English multiple-choice subset of InfiniteBench designed for 100K+ token context evaluation, requiring retrieval and reasoning over very long documents.

Multi-needle retrieval benchmark extending the 'needle in a haystack' test to require simultaneous retrieval of multiple facts embedded in long contexts.

Flexible benchmark testing effective context utilization across 13 diverse long-context tasks including retrieval, multi-hop reasoning, and aggregation at configurable lengths.

8,500 grade school math word problems requiring 2–8 steps of arithmetic reasoning. Widely used for evaluating chain-of-thought reasoning; top models now near 100%.

12,500 challenging competition mathematics problems from AMC, AIME, and other high-school math competitions across 7 difficulty levels.

250 GSM8K problems translated into 10 typologically diverse languages to evaluate multilingual math reasoning.

Visual language model benchmark for traffic anomaly understanding from surveillance video footage, using question-answering format to evaluate visual reasoning in transportation contexts.

PhD-level multiple-choice questions in biology, physics, and chemistry designed so that even experts with internet access score below 67%. The Diamond subset (198 questions) is the hardest variant used in most frontier model evaluations.

Reading comprehension benchmark requiring numerical reasoning, counting, sorting, and set operations over paragraphs.

Grade-school science multiple-choice questions partitioned into Easy and Challenge (hard) sets. ARC-Challenge is the standard evaluation variant.

Commonsense sentence-completion benchmark using adversarially filtered wrong answers. Top LLMs now exceed 95% accuracy.

Extended ANLS metric for evaluating visual question answering on documents, supporting free-form answer evaluation beyond exact match.

817 questions spanning 38 categories where humans commonly hold misconceptions. Measures whether models give truthful answers or reproduce popular falsehoods.

44,000 adversarially filtered Winograd schema problems for commonsense reasoning, created at scale to reduce dataset biases.

The easy partition of ARC with 7,787 grade-school science multiple-choice questions answerable without complex reasoning.

15,942 yes/no reading comprehension questions derived from real Google searches paired with Wikipedia passages.

12,102 commonsense reasoning multiple-choice questions crowdsourced using ConceptNet to target implicit world knowledge.

8,952 bilingual (Chinese/English) math and physics Olympiad problems from national/international competitions, testing advanced scientific reasoning with precise symbolic answers.

2,612 visual math problems across 5 versions (text dominant, text-lite, text-only, vision-intensive, vision-only) evaluating multimodal mathematical reasoning.

Abstraction and Reasoning Corpus (ARC-AGI) with visual grid puzzles testing core reasoning and novel problem-solving. ARC Prize 2024-2025 competitions track annual progress. NOTE: slug contains invalid characters '/' — recommend renaming to 'arc-prize'.

Dynamic adversarial data collection platform generating ever-harder NLU benchmarks through human-and-model-in-the-loop annotation to prevent contamination.

274,000 machine-generated toxic and benign statements about 13 minority groups, using adversarial classifier-in-the-loop decoding for implicit hate speech evaluation.

100,000 naturally occurring web text prompts for measuring the propensity of language models to generate toxic continuations using the Perspective API.

1,508 crowdsourced sentence pairs measuring stereotypical bias in masked language models across 9 categories including race, gender, and religion.

Ambiguous context subset of the Bias Benchmark for QA (BBQ), measuring how models respond to social bias questions when context is underspecified.

Disambiguated context subset of BBQ that tests whether additional clarifying context reduces biased model predictions.

720 Winograd-style schemas testing gender bias in pronoun resolution, checking whether models associate professions with gender stereotypes.

Type 1 (syntactic) Winobias coreference schemas measuring occupational gender bias where syntactic cues are ambiguous.

Type 2 (semantic) Winobias coreference schemas measuring occupational gender bias where gender-revealing pronouns provide semantic cues.

Berkeley Function Calling Leaderboard (BFCL) evaluating LLM ability to correctly call functions/APIs with proper arguments across diverse domains. Currently on v3.

Function calling benchmark from NexusFlow evaluating LLM tool use across zero-shot multi-function scenarios with nested function calls.

Adversarially collected NLI dataset across 3 rounds of increasing difficulty, designed to fool current models and expose failure modes.

11,500+ vision-language questions spanning 30 disciplines across six core areas (art, business, science, health, humanities, tech). Evaluates college-level multimodal reasoning.

Mathematical reasoning benchmark combining visual contexts (charts, geometry diagrams, scientific figures) from 28 existing datasets with 6,141 total problems.

Medical question answering benchmark based on USMLE licensing exams with 12,723 questions across US, Mainland China, and Taiwan medical boards.

5,957 elementary science multiple-choice questions requiring both open-book facts and broader commonsense knowledge to answer.

16,000 physical commonsense reasoning questions asking models to choose between two physical solutions for everyday tasks.

38,000 multiple-choice questions about everyday social situations requiring inference about emotions, motivations, and social norms.

1,000 causal reasoning questions asking models to choose the more plausible cause or effect for everyday scenarios.

Long-range language modeling benchmark predicting the last word of ~2,600 passages that require understanding broad discourse context.

28,000 reading comprehension passages with multiple-choice questions from Chinese middle and high school English exams.

100,000+ reading comprehension questions on 500+ Wikipedia articles. SQuAD 2.0 (2018) added unanswerable questions. Both versions remain widely cited.

98,407 conversational QA turns from 7,354 Wikipedia dialogues requiring multi-turn reasoning and follow-up question answering.

307,373 real Google search queries answered by extracting spans from Wikipedia, with short answer, long answer, and no-answer categories.

433,000 hypothesis-premise pairs across 10 genres (fiction, telephone, travel, etc.) for textual entailment classification.

13,679 multiple-choice science questions across physics, chemistry, biology, and earth science, each paired with a supporting paragraph.

Counterfactual reasoning benchmark testing LLMs on hypothetical scenarios and their logical implications.

Benchmark for generating clinical notes from doctor-patient conversation transcripts to evaluate ambient clinical intelligence systems.

Large-scale machine reading comprehension and passage ranking benchmark from real Bing search queries, widely used for information retrieval evaluation.

Query-focused meeting summarization benchmark with 1,808 query-summary pairs across parliamentary, product, and academic meeting transcripts.

Anthropic's alignment evaluation covering three core dimensions: helpfulness, honesty, and harmlessness, via human preference comparisons between model responses.

Microsoft's framework for automated measurement of responsible AI harms in generative AI applications across fairness, reliability, safety, and privacy dimensions.

10 code intelligence tasks across code completion, code translation, code summarization, and code search for multiple programming languages.

LLM-as-a-judge evaluation paradigm using strong models (GPT-4) to score responses, enabling scalable assessment of open-ended generation quality.

Unified multi-dimensional automatic evaluation framework for LLMs assessing helpfulness, engagement, safety, and relevance in a single pass.

Fine-tuned language models trained to be scalable judges for open-ended LLM evaluation, achieving high agreement with human preferences.

LLM evaluation framework using rubric-based scoring with reference materials, enabling fine-grained absolute/relative assessment without GPT-4 dependency.

Visual question answering on 12,767 document images (scanned forms, invoices, tables) requiring understanding of text layout and document structure.

Comprehensive Chinese-language evaluation suite with 13,948 multiple-choice questions across 52 disciplines and 4 difficulty levels (middle school to professional).

67-task Chinese MMLU equivalent covering 11,528 questions across natural science, social science, STEM, and Chinese-specific cultural knowledge.

Evaluation benchmark using questions from China's college entrance examination (Gaokao) across math, language arts, science, and social studies.

Multimodal extension of GAOKAO with text+image questions from Chinese college entrance exams, evaluating vision-language model capabilities.