DBRX
About
The DBRX family of large language models (LLMs), developed by Databricks, includes DBRX Base and DBRX Instruct, both of which have been released under an open license. These transformer-based, decoder-only models employ a distinguished fine-grained mixture-of-experts (MoE) architecture. This approach leverages a larger number of smaller expert networks, enhancing the quality of output compared to models with fewer, larger experts. Trained on an extensive dataset of 12 trillion tokens encompassing text and code, these models showcase high performance with the capability to handle up to 32,000 tokens of context. DBRX Instruct is specially optimized for few-turn interactions, making it suitable for efficient conversational applications. The DBRX models not only excel in efficiency during training and inference but also outperform many leading closed and open-source models in various benchmarks. They are accessible for download on Hugging Face and integrated through Databricks' Foundation Model APIs.