Concepts & capability filters
preprocessing
Tokenization
Tokenization is the process of breaking down input text into smaller units called tokens (e.g., words, subwords, or characters) that the model can process numerically.
- Category
- preprocessing
- Difficulty
- Not classified
- Aliases
- None tracked
- Last reviewed
- 2026-07-02
Key facts
- It is the first step in preparing data for LLMs, affecting vocabulary size and sequence length.