MetaMath Llemma 7B
About
MetaMath-Llemma-7B is a fine-tuned large language model designed specifically for mathematical reasoning, building upon the Llemma-7B architecture. It showcases notably enhanced performance over similar-sized open-source models in solving mathematical problems, thanks to a novel question bootstrapping method that enriches its training data with diverse mathematical questions. Trained on the comprehensive MetaMathQA dataset, which is derived from GSM8K and MATH benchmarks, the model achieves impressive scores: Pass@1 scores of 69.2% on GSM8K and 30.0% on MATH. Despite its achievements, it remains outperformed by closed-source models like GPT-4, pointing to avenues for further improvement. Additionally, its reliance on datasets augmented with tools like ChatGPT 3.5 may introduce certain biases. Nevertheless, its open-source nature allows for ongoing research and improvements by the AI community.