activeMultimodal

CharXiv

Metric: Score (higher is better)

Chart understanding benchmark with descriptive and reasoning questions

Models ranked

7

tracked on this benchmark

Score band

89.0 – 82.5

best → lowest tracked

Snapshot trend

+1.55

Oct 19 → Oct 26 · 2 models

Leaderboard

Tracked models ranked by Score (higher is better).

Compare candidates
#Model variant and provenanceScore
1
Version: CharXiv RQ; no tools; max reasoning effortHarness: Not recordedEvaluator: Not recordedObserved: Oct 23, 2026Confidence: Not recordedSource

Notes: Moonshot self-reported launch table, row 'CharXiv (RQ)', column 'Kimi K3.1 Pro (max)'. Settings: reasoning effort max, temperature 1.0, top-p 0.95 (blog footnote).

89.0
2

Configuration: Muse Spark 1.1 (xhigh)

Version: CharXiv Reasoning (1,000 val Q)Harness: Static eval; gpt-oss-120b (high) LLM judgeEvaluator: MetaObserved: Oct 17, 2026Confidence: self-reportedSource

Notes: Muse Spark 1.1 model page benchmark table (raw HTML parsed). Methodology: evaluation report https://ai.meta.com/static-resource/muse-spark-1-1-evaluation-report/ (2026-07-09). All Muse Spark 1.1 results run via Meta Model API at xhigh reasoning effort (report §5.1, p.100-101). Report §5.2.4. Caveat: §5.1 says multimodal evals run with server-side code execution where applicable; page does not mark CharXiv as w/ tools. Retrieved 2026-10-17.

88.4
3

Configuration: Seed2.1-Pro

Version: CharXiv-RQ (reasoning), no tools, Pass@1Harness: Static eval (no agent harness); ByteDance Seed internal evalEvaluator: ByteDance SeedObserved: Oct 17, 2026Confidence: self-reportedSource

Notes: Seed2.1 Model Card PDF Table 9 (metric Pass@1); value also transcribed from blog chart images https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity (downloaded to img/seed21-*.png) and matches the card. With-tools 86.4 and CharXiv-DQ 95.5 also reported (not packed). Retrieved 2026-10-17.

85.4
4
Version: CharXiv RQ; three-run mean; max reasoning effortHarness: Not recordedEvaluator: Not recordedObserved: Jul 16, 2026Confidence: Not recordedSource

Notes: Model variant: Kimi K3 (max reasoning effort). Benchmark variant: CharXiv RQ, three-run mean. Harness/evaluator not disclosed by Moonshot. Source posture: Moonshot/Kimi self-reported launch table; confidence: medium because the value is vendor-reported and not independently reproduced. Recommended seed value: 84.8.

84.8
5

Configuration: Qwen3.8-Flash

Version: CharXiv RQ (reasoning), without code interpreterHarness: Static eval, no code interpreterEvaluator: Qwen (Alibaba)Observed: Oct 26, 2026Confidence: self-reportedSource

Notes: Qwen Team tech report 'Qwen3.8-Omni: Towards Native Omni-Modal Agents' (arXiv 2609.25611v1, 2026-09-22), Table 3 column 'Qwen3.8-Flash', row 'CharXiv (RQ)' without CI: 84.6 (w/ CI row not packed). Fixed step-by-step \boxed{} prompt for Qwen models (footnote 3). Identity: the report's 'Qwen3.8-Flash' column and the QwenCloud qwen3.8-flash docs page carry the same values as the Qwen3.8-Flash-Next blog/HF card; Qwen's blog says Qwen3.8-Flash is the production served version (1M context, built-in tools) while these evals used a 256K context. Retrieved 2026-10-26.

84.6
6

Configuration: Qwen3.8-Omni-Flash

Version: CharXiv RQ (reasoning), without code interpreterHarness: Static eval, no code interpreterEvaluator: Qwen (Alibaba)Observed: Oct 26, 2026Confidence: self-reportedSource

Notes: Qwen Team tech report 'Qwen3.8-Omni: Towards Native Omni-Modal Agents' (arXiv 2609.25611v1, 2026-09-22), Table 3 column 'Qwen3.8-Omni-Flash', row 'CharXiv (RQ)' without CI: 83.5 (w/ CI row not packed). Fixed step-by-step \boxed{} prompt for Qwen models (footnote 3). Retrieved 2026-10-26.

83.5
7

Configuration: MiniMax M3.1 Flash

Version: CharXiv RQ (reasoning)Harness: Static eval; GPT-5.5 judgeEvaluator: MiniMaxObserved: Oct 19, 2026Confidence: self-reportedSource

Notes: MiniMax M3.1 Flash launch blog (2026-10-19) main benchmark table image m31-flash-en-2 (webp converted to png; values transcribed by eye from table) incl. footnotes. Footnote: GPT-5.5 as judge. Retrieved 2026-10-19.

82.5

How to read this benchmark

This benchmark scores models where higher is better. Use scores for directional filtering and shortlisting, not universal quality ranking; then validate pricing, context window, provider availability, and fit for your workload.

Trust this score when

  • There is a fresh timestamped snapshot (or multiple snapshots) for this benchmark.
  • The model list covers the same version family you can actually deploy today.
  • Top candidates overlap with your required routing and feature requirements.

Be cautious when

  • There is only one benchmark snapshot or the dataset appears stale.
  • The benchmark metric direction is opposite of your decision objective.
  • The score difference between options is narrow and likely within implementation variance.

Related benchmarks

Last reviewed: May 20, 2026

Resources