Llemma Models by EleutherAI
Last refreshed 2026-05-19. Next refresh: weekly.
Details
Capabilities
About
Llemma is a family of open-access large language models (LLMs) designed to specialize in mathematical reasoning. Developed by EleutherAI, these models were initialized using Code Llama weights and trained on the Proof-Pile-2 dataset, comprising a vast 55 billion unique tokens of mathematical and scientific documents. This extensive training allows Llemma models to excel in chain-of-thought mathematical reasoning and effectively use computational tools like Python and formal theorem provers. Available in both 7-billion and 34-billion parameter variants, the Llemma models, particularly the larger one, outperform other LLMs of similar size on a range of mathematical benchmarks. The Llemma project's open-source approach facilitates ongoing research and advancements in mathematical reasoning with LLMs 23.
Decision facts
- Best fit
- mathematicsstructured outputscoding
- Capability starting point
- Llemma 7B with 4k context and structured outputs
- Lowest tracked input
- Not tracked
- Closest related family
- InternLM2-Math
Current Variants
Use-when guidance is based on each model's tracked capabilities, context window, release date, and replacement status.
Use when the workload needs 4k context and 34B parameters.
Use when the workload needs 4k context, 7B parameters, and structured outputs.
| Model | Use when | Released | Signals | Status |
|---|---|---|---|---|
| Llemma 34B | Use when the workload needs 4k context and 34B parameters. | 2023-09 | 4k context34B parameters | Current |
| Llemma 7B | Use when the workload needs 4k context, 7B parameters, and structured outputs. | 2023-09 | 4k context7B parametersstructured outputs | Current |
Release Timeline
1 release groupSpecifications(2 models)
| Model | Released | Context | Parameters | Structured Outputs |
|---|---|---|---|---|
| Llemma 34B | 2023-09 | 4k | 34B | No |
| Llemma 7B | 2023-09 | 4k | 7B | Yes |
