RedPajama INCITE 3B

About

RedPajama-INCITE 3B is an open-source large language model designed by Together Computer, featuring 2.8 billion parameters and employing the Pythia architecture, a variant of GPT-3. Utilizing fully dense attention layers, FlashAttention, Rotary Positional Embeddings, parallelized attention, and untied embedding matrices, it excels in various natural language processing tasks like question answering, text generation, summarization, and conversational AI. Trained on a diverse 1.2 trillion token dataset, it includes content primarily in English alongside 20 other languages. Despite its advantages in accessibility and deployment due to its relatively small size, it exhibits limitations compared to larger models and requires responsible use.

Capabilities