Nemotron 3 Super-120B-A12B

NVIDIA Nemotron-3 Super-120B-A12B is a hybrid Mamba-Transformer mixture-of-experts language model developed by NVIDIA Research, targeting agentic reasoning, long-document processing, and multi-step task planning. The model has 120 billion total parameters with 12 billion activated parameters per token, achieved through a Latent Mixture-of-Experts (LatentMoE) architecture that interleaves Mamba-2 state-space model layers with sparse MoE layers and select self-attention layers. Mamba-2 processes sequences in linear time with respect to sequence length, enabling efficient handling of very long contexts.

The model supports a context window of up to 1 million tokens per NVIDIA's technical report, with the Mamba-2 layers providing linear-time scaling. NVIDIA pretrained the model on 25 trillion tokens using NVFP4 (NVIDIA's 4-bit floating-point format optimized for Blackwell GPUs). Multi-Token Prediction (MTP) layers accelerate text generation throughput. The LatentMoE design routes 4 experts at the compute cost of 1, improving intelligence-per-FLOP efficiency. Note that API providers may enforce shorter context limits (Fireworks AI lists 262K for the NVFP4 variant).

The base BF16 weights and NVFP4 variant are available on Hugging Face (nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16) under NVIDIA's open model license. API access is available through NVIDIA NIM, Fireworks AI, DeepInfra, and OpenRouter. The model is designed for NVIDIA Blackwell hardware but can run in BF16 on earlier GPU generations.