CharXiv
Chart understanding benchmark with descriptive and reasoning questions
Models ranked
7
tracked on this benchmark
Score band
89.0 – 82.5
best → lowest tracked
Snapshot trend
+1.55
Oct 19 → Oct 26 · 2 models
Leaderboard
Tracked models ranked by Score (higher is better).
Notes: Moonshot self-reported launch table, row 'CharXiv (RQ)', column 'Kimi K3.1 Pro (max)'. Settings: reasoning effort max, temperature 1.0, top-p 0.95 (blog footnote).
Configuration: Muse Spark 1.1 (xhigh)
Notes: Muse Spark 1.1 model page benchmark table (raw HTML parsed). Methodology: evaluation report https://ai.meta.com/static-resource/muse-spark-1-1-evaluation-report/ (2026-07-09). All Muse Spark 1.1 results run via Meta Model API at xhigh reasoning effort (report §5.1, p.100-101). Report §5.2.4. Caveat: §5.1 says multimodal evals run with server-side code execution where applicable; page does not mark CharXiv as w/ tools. Retrieved 2026-10-17.
Configuration: Seed2.1-Pro
Notes: Seed2.1 Model Card PDF Table 9 (metric Pass@1); value also transcribed from blog chart images https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity (downloaded to img/seed21-*.png) and matches the card. With-tools 86.4 and CharXiv-DQ 95.5 also reported (not packed). Retrieved 2026-10-17.
Notes: Model variant: Kimi K3 (max reasoning effort). Benchmark variant: CharXiv RQ, three-run mean. Harness/evaluator not disclosed by Moonshot. Source posture: Moonshot/Kimi self-reported launch table; confidence: medium because the value is vendor-reported and not independently reproduced. Recommended seed value: 84.8.
Configuration: Qwen3.8-Flash
Notes: Qwen Team tech report 'Qwen3.8-Omni: Towards Native Omni-Modal Agents' (arXiv 2609.25611v1, 2026-09-22), Table 3 column 'Qwen3.8-Flash', row 'CharXiv (RQ)' without CI: 84.6 (w/ CI row not packed). Fixed step-by-step \boxed{} prompt for Qwen models (footnote 3). Identity: the report's 'Qwen3.8-Flash' column and the QwenCloud qwen3.8-flash docs page carry the same values as the Qwen3.8-Flash-Next blog/HF card; Qwen's blog says Qwen3.8-Flash is the production served version (1M context, built-in tools) while these evals used a 256K context. Retrieved 2026-10-26.
Configuration: Qwen3.8-Omni-Flash
Notes: Qwen Team tech report 'Qwen3.8-Omni: Towards Native Omni-Modal Agents' (arXiv 2609.25611v1, 2026-09-22), Table 3 column 'Qwen3.8-Omni-Flash', row 'CharXiv (RQ)' without CI: 83.5 (w/ CI row not packed). Fixed step-by-step \boxed{} prompt for Qwen models (footnote 3). Retrieved 2026-10-26.
Configuration: MiniMax M3.1 Flash
Notes: MiniMax M3.1 Flash launch blog (2026-10-19) main benchmark table image m31-flash-en-2 (webp converted to png; values transcribed by eye from table) incl. footnotes. Footnote: GPT-5.5 as judge. Retrieved 2026-10-19.
How to read this benchmark
This benchmark scores models where higher is better. Use scores for directional filtering and shortlisting, not universal quality ranking; then validate pricing, context window, provider availability, and fit for your workload.
Trust this score when
- There is a fresh timestamped snapshot (or multiple snapshots) for this benchmark.
- The model list covers the same version family you can actually deploy today.
- Top candidates overlap with your required routing and feature requirements.
Be cautious when
- There is only one benchmark snapshot or the dataset appears stale.
- The benchmark metric direction is opposite of your decision objective.
- The score difference between options is narrow and likely within implementation variance.
Related benchmarks
Last reviewed: May 20, 2026