Smaug 34B

About

Smaug-34B-v0.1 is an advanced large language model developed by Abacus.AI, evolving from the Bagel-34B-v0.2 with 34.4 billion parameters. It employs the LlamaForCausalLM architecture and is accessible via Hugging Face. This model is fine-tuned using DPO-Positive, a novel technique enhancing conventional optimization methods, leading to improved outcomes on tasks with minimal data diversity. It leverages datasets like ARC and HellaSwag, but concerns exist regarding overfitting due to dataset similarities with benchmark tests. Smaug-34B-v0.1 scores an impressive average of 77.29% on several benchmarks, although its performance varies depending on contamination levels, and the absence of DPOP code may hinder reproducibility efforts.

Capabilities