StarCoder 2
About
StarCoder 2 is an advanced family of open large language models (LLMs) adept at code generation, developed collaboratively by BigCode, Hugging Face, and NVIDIA. This model family includes variants with 3 billion, 7 billion, and 15 billion parameters. Trained on The Stack v2, a colossal dataset over 4 trillion tokens and more than 600 programming languages, StarCoder 2 is built for high performance. It leverages innovative techniques like Grouped Query Attention, a 16,384-token context window, and a 4,096-token sliding window attention, employing the Fill-in-the-Middle objective during training. The StarCoder2-15B model, in particular, showcases exceptional capabilities, often matching or surpassing larger models. Available under the BigCode OpenRAIL-M v1 license, StarCoder 2 offers extensive potential for complex code-based applications134.