Striped Hyena
About
The Striped Hyena family of large language models (LLMs), developed by Together AI and collaborators, offers an innovative approach to model architecture with its hybrid design. This family integrates multi-head, grouped-query attention and gated convolutions within Hyena blocks, which distinguishes it from conventional transformer models. This unique structure enhances training and inference capabilities, particularly for handling long sequences. The family encompasses both base models like the StripedHyena-Hessian-7B (SH 7B) and chat models such as StripedHyena-Nous-7B (SH-N 7B), all optimized for efficiency with faster processing speeds and reduced memory usage during autoregressive tasks. Designed to handle prompt lengths up to 128,000 tokens, Striped Hyena models deliver performance competitive with top open-source transformers on a variety of benchmarks 46.