Llemma Models by EleutherAI
About
Llemma is a family of open-access large language models (LLMs) designed to specialize in mathematical reasoning. Developed by EleutherAI, these models were initialized using Code Llama weights and trained on the Proof-Pile-2 dataset, comprising a vast 55 billion unique tokens of mathematical and scientific documents. This extensive training allows Llemma models to excel in chain-of-thought mathematical reasoning and effectively use computational tools like Python and formal theorem provers. Available in both 7-billion and 34-billion parameter variants, the Llemma models, particularly the larger one, outperform other LLMs of similar size on a range of mathematical benchmarks. The Llemma project's open-source approach facilitates ongoing research and advancements in mathematical reasoning with LLMs 23.
Current Variants
Use-when guidance is derived from seed capabilities, context, release, and replacement fields.
Use when the workload needs 4k context and 34B parameters.
Use when the workload needs 4k context, 7B parameters, and structured outputs.
| Model | Use when | Released | Signals | Status |
|---|---|---|---|---|
| Llemma 34B | Use when the workload needs 4k context and 34B parameters. | 2023-09 | 4k context34B parameters | Current |
| Llemma 7B | Use when the workload needs 4k context, 7B parameters, and structured outputs. | 2023-09 | 4k context7B parametersstructured outputs | Current |
Release Timeline
1 release groupSpecifications(2 models)
| Model | Released | Context | Parameters | Structured Outputs |
|---|---|---|---|---|
| Llemma 34B | 2023-09 | 4k | 34B | No |
| Llemma 7B | 2023-09 | 4k | 7B | Yes |
Frequently Asked Questions
- What is Llemma used for?
- Llemma is used for mathematics, structured outputs, and coding. The family description and listed model capabilities point to those workloads as the best fit.
- How does Llemma compare to InternLM2-Math?
- Llemma by EleutherAI is strongest where you need mathematics, while InternLM2-Math by Intern-AI is the closest related family to check for mathematics. Llemma has 2 listed variants and reaches up to 4k context, while InternLM2-Math reaches up to 64k context, so compare the specs and pricing tables before choosing a production model.
- Which Llemma model should I use?
- If price is the main constraint, use the pricing table first because Llemma does not have complete provider pricing in the local data. For the most capable/latest local choice, evaluate Llemma 7B with 4k context and structured outputs.
