WizardLM-2 8x22B

WizardLM-2 8x22B is an instruction-tuned mixture-of-experts model released April 16, 2024 by the WizardLM research team at Microsoft. It is built on the Mistral 8x22B MoE backbone, which has approximately 141 billion total parameters with around 39 billion active parameters per token during inference. The context window is 65,536 tokens. The model is released under the Apache 2.0 license.

The training methodology uses Evol-Instruct, a technique that generates increasingly complex synthetic instruction-response pairs through iterative AI-driven evolution, combined with an alignment framework called AI Align AI. This approach produces a model optimized for complex multi-step instruction adherence, multilingual conversations, logical reasoning, and agentic task execution. At release, WizardLM-2 8x22B claimed to outperform GPT-4 (March 2024 version) on MT-Bench while exceeding all then-existing open-source alternatives on that benchmark.

WizardLM-2 8x22B is the largest and most capable member of the WizardLM-2 family, alongside 8x7B and 7B variants. It is available through DeepInfra, OctoAI, OpenRouter, Novita AI, and LeptonAI. The model is appropriate for applications requiring detailed instruction adherence, multi-step reasoning chains, and multilingual coverage in an open-weight MoE framework.