Benchmarks

Chatbot Arena

Chatbot Arena

Compares chatbots based on crowdsourced human preferences

MT-Bench

MT-Bench

Compares models based on human preferences

HumanEval

HumanEval

Evaluates code generation abilities

MBPP

Mostly Basic Programming Problems

Tests code generation in Python

HumanEval+

HumanEval+

MBPP+

Mostly Basic Programming Problems+

Spider

Spider

SQL generation

ToolTalk

ToolTalk

Evaluates Tool LLMs in a conversational setting

BBH

BIG-Bench Hard

Evaluates models on a mix of challenging tasks

WildBench

WildBench

Evaluates models on a diverse set of tasks

AGIEval

Artificial General Intelligence Eval

MMLU

Massive Multitask Language Understanding

Tests LLMs on undergraduate level knowledge across 57 subjects

IFEval

Instruction-Following Evaluation

MMLU PRO

MMLU PRO

TriviaQA

TriviaQA

ZeroSCROLLS/QuALITY

ZeroSCROLLS/QuALITY

InfiniteBench/En.MC

InfiniteBench/En.MC

NIH/Multi-needle

NIH/Multi-needle

RULER

RULER

Effective context length

GSM8K

Grade School Math 8K

Tests grade school math problem solving

MATH

Mathematics Aptitude Test of Heuristics

Evaluates math word problem-solving abilities

MGSM

Multilingual Grade School Math

Tests multilingual math capabilities

GPQA

Google-Proof Q&A

Evaluates graduate level reasoning capabilities

DROP

Discrete Reasoning Over Paragraphs

Tests reasoning over text

ARC

AI2 Reasoning Challenge

Tests knowledge-based question answering

HellaSwag

HellaSwag

Evaluates common sense reasoning

ANLS*

Average Normalized Levenshtein Similarity*

Evaluates visual question answering on documents

TruthfulQA

TruthfulQA

Assesses the truthfulness of model responses

WinoGrande

WinoGrande

Tests common sense reasoning

ARC-E

AI2 Reasoning Challenge-Easy

BoolQ

Boolean Questions

CommonsenseQA

CommonsenseQA

ToxiGen

ToxiGen

RealToxicity

RealToxicity

CrowS-Pairs

CrowS-Pairs

BBQ Ambig

BBQ Ambig

BBQ Disambig

BBQ Disambig

Winogender

Winogender

Winobias 1_2

Winobias 1_2

Winobias 2_2

Winobias 2_2

BFCL

BFCL

Nexus

Nexus

ANLI

Adversarial Natural Language Inference

MMMU

Massive Multi-discipline Multimodal Understanding

Tests math and reasoning skills on multimodal data

MathVista

MathVista

Tests mathematical reasoning abilities

MedQA

MedQA

OpenBookQA

OpenBookQA

PIQA

Physical Interaction: Question Answering

SocialIQA

Social Intelligence QA

COPA

Choice of Plausible Alternatives

LAMBADA

LAnguage Modeling Broadened to Account for Discourse Aspects

RACE

ReAding Comprehension Dataset From Examinations

SQuAD

Stanford Question Answering Dataset

QuAC

Question Answering in Context

NaturalQ

Natural Questions

MultiNLI

Multi-Genre Natural Language Inference

SciQ

SciQ

CRASS

Counterfactual Reasoning Assessment

ACI-BENCH

Ambient Clinical Intelligence Benchmark

MS-MARCO

MAchine Reading COmprehension Dataset

QMSum

Query-based Multi-domain Meeting Summarization

HHH

Helpfulness, Honesty, Harmlessness

RAI

Responsible AI

CodeXGLUE

CodeXGLUE

LLM Judge

LLM Judge

LLM-Eval

LLM-Eval

JudgeLM

JudgeLM

Prometheus

Prometheus

DocVQA

DocVQA

C-Eval

Chinese Evaluation Suite

C-Eval is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels.

CMMLU

Chinese Massive Multitask Language Understanding

CMMLU is a comprehensive evaluation benchmark specifically designed to evaluate the knowledge and reasoning abilities of LLMs within the context of Chinese language and culture.

GAOKAO

GAOKAO

An evaluation benchmark based on a dataset of Chinese college entrance examination questions.

GAOKAO-MM

GAOKAO-MM

A multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO).