JevBench public-231active

JevBench public-231: JevBench public 231 (self-run accuracy)

Metric: Accuracy (%) (higher is better)Introduced: 2026

Accuracy on the 231 public JevBench items (easy 48, original/standard 72, hard 111 in datasets/public/) run with the open JevBench CLI at github.com/fstandhartinger/jevbench. Scores are vendor-reported self-runs of that public split, not a composite leaderboard rank.

Models ranked

3

tracked on this benchmark

Score band

88.7 – 76.2

best → lowest tracked

Snapshot trend

—

need ≥2 snapshots

Leaderboard

Tracked models ranked by Accuracy (%) (higher is better).

Compare candidates
#Model variant and provenanceScore
1
H2O-Lightning-4B
Version: public-231Harness: Not recordedEvaluator: Not recordedObserved: Oct 15, 2026Confidence: Not recordedSource

Notes: Vendor-reported. H2O.ai HF card: JevBench text, 231 public items via JevBench CLI: 205/231 (88.7%); easy 48/48, standard 71/72, hard 86/111. Model v1.1.

88.7
2
Torchcast Decision 12B
Version: public-231Harness: Not recordedEvaluator: Not recordedObserved: Oct 15, 2026Confidence: Not recordedSource

Notes: Vendor-reported. Torchcast HF card: author-run, not an official JevBench score: 203/231 (87.88%) on JevBench v1.4 public set, CLI bb05a335. Card states public JevBench results influenced checkpoint and serving-configuration selection.

87.9
3
Strands Decider 2B
Version: public-231Harness: Not recordedEvaluator: Not recordedObserved: Oct 15, 2026Confidence: Not recordedSource

Notes: Vendor-reported. Strands Labs HF card: JevBench public, window 4096: 176/231 (76.19%); six-seed mean 175.0. Brier 0.323, ECE 0.064.

76.2

How to read this benchmark

This benchmark scores models where higher is better. Use scores for directional filtering and shortlisting, not universal quality ranking; then validate pricing, context window, provider availability, and fit for your workload.

Trust this score when

  • There is a fresh timestamped snapshot (or multiple snapshots) for this benchmark.
  • The model list covers the same version family you can actually deploy today.
  • Top candidates overlap with your required routing and feature requirements.

Be cautious when

  • There is only one benchmark snapshot or the dataset appears stale.
  • The benchmark metric direction is opposite of your decision objective.
  • The score difference between options is narrow and likely within implementation variance.

Last reviewed: Oct 15, 2026

Resources

Website