Rotary Position Embedding

RoPE (Rotary Position Embedding) encodes token positions by rotating query and key vectors, giving Transformers relative-position awareness that extrapolates better to longer sequences. It is a common choice in modern open-weight LLMs and underpins many of the long-context implementations you would evaluate.