Using StarCoder2 15B on DeepInfra
Implementation guide · StarCoder 2 · ServiceNow Research
Quick Start
- 1
- 2Use the DeepInfra SDK or REST API to call
starcoder2-15b— see the documentation for request format. - 3
Code Examples
pip install openaiDEEPINFRA_API_KEYstarcoder2-15bDeepInfra uses "organization/model-name" format, e.g. "meta-llama/Meta-Llama-3-8B-Instruct" or "mistralai/Mistral-7B-Instruct-v0.3". See the DeepInfra model catalog for exact IDs.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["DEEPINFRA_API_KEY"],
base_url="https://api.deepinfra.com/v1/openai"
)
response = client.chat.completions.create(
model="starcoder2-15b",
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)About DeepInfra
DeepInfra offers serverless AI inference with a simple API, supporting hundreds of models across text generation, embeddings, and more. Pay-per-token pricing with no upfront commitments.
DeepInfra is a cloud inference platform offering cost-effective access to open-source AI models. It provides serverless inference for leading models from Meta, Mistral, Alibaba, and others with competitive token-based pricing.
Pricing on DeepInfra
| Type | Price (per 1M) |
|---|---|
| Input tokens | $0.20 |
| Output tokens | $0.60 |
Capabilities
About StarCoder2 15B
StarCoder2-15B is a sophisticated large language model, expertly crafted for code generation and understanding. Developed by the BigCode project, it features 15 billion parameters and is trained on The Stack v2, a vast dataset of over 4 trillion tokens from more than 600 programming languages. Its advanced transformer decoder architecture, equipped with a grouped-query and sliding window attention mechanism and a Fill-in-the-Middle training objective, allows a context window of 16,384 tokens. In addition to generating and completing code, the model excels in tasks like code summarization and retrieving relevant snippets through natural language queries. The training leveraged NVIDIA's NeMo framework and the Eos Supercomputer, while usage is governed by the BigCode Open RAIL-M license, supporting royalty-free and commercial use.