Using Dolphin 2.6 Mixtral 8x7B on Lepton AI API
Implementation guide · Dolphin · Cognitive Computations
Quick Start
- 1
- 2Use the Lepton AI API SDK or REST API to call
dolphin-2.6-mixtral-8x7b— see the documentation for request format. - 3
Code Examples
About Lepton AI API
Lepton AI is a comprehensive cloud-native platform designed to simplify the development and deployment of AI applications. It offers a user-friendly interface that allows developers to build models natively in Python, eliminating the need for complex containerization or Kubernetes expertise. The platform supports local debugging, enabling users to test their models before deployment with a simple command. With a flexible API for easy integration into various applications and support for heterogeneous hardware, Lepton AI optimizes performance based on specific application needs. This flexibility allows for efficient scaling, accommodating workloads that can expand up to 1TB of memory. The platform provides a robust set of tools and infrastructure to enhance AI workflows. Its cloud-native architecture supports high-performance computing, featuring smart scheduling and dynamic batching to minimize downtime. Lepton AI enables continuous deployment through GitHub integration, facilitating rapid iteration and scaling of AI applications. The platform also includes built-in monitoring, logging, and autoscaling capabilities, ensuring that applications remain responsive and efficient in production environments. With these features, Lepton AI streamlines the entire AI development process, from model creation to deployment and maintenance, making it accessible for organizations of various sizes looking to innovate with AI technologies.
Lepton AI is building a scalable and efficient AI Application platform. Their platform aims to simplify the development and deployment of AI applications, making it easier for businesses to leverage artificial intelligence technologies. The company focuses on providing tools and infrastructure to streamline AI workflows, enabling faster development cycles and more efficient resource utilization. While specific details about their platform's features are not provided in the context, Lepton AI's mission is to make AI application development more accessible and efficient for developers and businesses alike.
Pricing on Lepton AI API
| Type | Price (per 1M) |
|---|---|
| Input tokens | $0.30 |
| Output tokens | $0.30 |
Capabilities
No model capability flags are currently sourced.
About Dolphin 2.6 Mixtral 8x7B
Dolphin 2.6 Mixtral 8x7B is a large language model fine-tuned from the Mixtral-8x7B base, known for its robust coding abilities and high compliance with user prompts. Despite not being tuned with Direct Preference Optimization, it performs exceptionally well in coding tasks due to extensive training with coding datasets, including MagiCoder. The model's architecture features a context window reduced to 16k, and training was carried out using techniques like qLoRA. However, it is uncensored, exposing potential ethical concerns and prompting caution for deployment without additional safeguards. Quantized versions are available to accommodate different hardware needs, and users may encounter variation in performance with larger context windows.