Grok-1
About
Grok-1, created by xAI, is a formidable 314-billion parameter Mixture-of-Experts (MoE) language model. It boasts a sophisticated architecture with 8 experts, leveraging 2 for each token input, spread across 64 layers and equipped with 48 attention heads per query. This vast model was trained from scratch using a specially crafted training stack based on JAX and Rust, finishing its pre-training phase by October 2023. Released as a base model under the permissive Apache 2.0 license, its open-source framework allows both commercial and non-commercial applications, though it lacks fine-tuning for specific tasks. Benchmarks highlight Grok-1's superior reasoning on various tasks but recognize its potential for generating inaccuracies ("hallucinations"). Running on a local setup requires substantial hardware, including a multi-GPU system, for efficient performance.