Skywork (Kunlun)

About

Skywork AI, an AI research subsidiary of Kunlun Tech, has quickly established itself as a notable player in the domain of Artificial General Intelligence (AGI) and AI-generated content (AIGC) since its inception in 2023. The Beijing-based company focuses on advancing large language models (LLMs) and has developed the Skywork-13B family, a bilingual foundation model trained on over 3.2 trillion tokens of English and Chinese text. This achievement marks it as one of the most extensively trained and openly published LLMs of its scale, showcasing notable performance in Chinese language modeling across diverse domains. A distinguishing feature of Skywork AI is its commitment to openness and collaboration. It has made a significant portion of its models and training data available to the public, encouraging community participation and innovation in AI research. This approach starkly contrasts with the more closed strategies typical of some competitors, promoting accessibility and shared progress within the AI community. Technologically, Skywork AI excels in developing specialized LLMs optimized for various applications, such as chat, mathematical problem-solving, and multimodal interactions. In an innovative stride, the company introduced the SkyPile corpus, the largest publicly accessible pre-training corpus of high-quality Chinese web text, comprising over 150 billion tokens. Skywork's two-stage training methodology, involving general-purpose and domain-specific refinement, contributes to the superior performance of its models. Beyond LLMs, Skywork AI's research interests span multi-modal content generation and understanding, with a focus on diffusion models for media generation. Additionally, its novel leakage detection method addresses the crucial issue of test data contamination in LLM training. Despite its broad research ambitions, details on Skywork's financial standing or specific investor contributions remain sparse, leaving an element of mystery around its financial operations.