Using Llama 3.1 70B Instruct on Vercel AI Gateway
Implementation guide · Llama 3.1 · AI at Meta
Quick Start
- 1
- 2Use the Vercel AI Gateway SDK or REST API to call
meta/llama-3.1-70b— see the documentation for request format. - 3
Code Examples
pip install openaiAI_GATEWAY_API_KEYmeta/llama-3.1-70bcreator/model-name e.g. kwaipilot/kat-coder-pro-v2
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["AI_GATEWAY_API_KEY"],
base_url="https://ai-gateway.vercel.sh/v1"
)
response = client.chat.completions.create(
model="meta/llama-3.1-70b",
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)About Vercel AI Gateway
Vercel AI Gateway is a unified AI proxy providing a single OpenAI-compatible API endpoint to 275+ models from 25+ providers including Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, xAI, Alibaba, Amazon, ByteDance, Cohere, MiniMax, MoonshotAI, KwaiPilot, Black Forest Labs, Recraft, Voyage AI, NVIDIA, and more. Pricing is pass-through at provider list rates with zero markup. Includes $5/month free tier; paid is pay-as-you-go. Features: automatic provider fallbacks, unified observability, streaming, tool use, vision, embeddings, image/video generation, BYOK mode. Integrates via @ai-sdk/gateway package or plain model ID strings in Vercel AI SDK. API details: API key via Authorization: Bearer <AI_GATEWAY_API_KEY>. Key from Vercel dashboard. Free $5/month credit; paid tier is provider list price with zero markup. BYOK (bring-your-own-key) also supported with no markup or fee. Model IDs use {provider-owner}/{model-name} — e.g., anthropic/claude-opus-4.6, openai/gpt-5.
Vercel AI Gateway is a unified AI proxy providing a single OpenAI-compatible API endpoint to 275+ models from 25+ providers including Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, xAI, Alibaba, Amazon, ByteDance, Cohere, MiniMax, MoonshotAI, KwaiPilot, Black Forest Labs, Recraft, Voyage AI, NVIDIA, and more. Pricing is pass-through at provider list rates with zero markup. Includes $5/month free tier; paid is pay-as-you-go. Features: automatic provider fallbacks, unified observability, streaming, tool use, vision, embeddings, image/video generation, BYOK mode. Integrates via @ai-sdk/gateway package or plain model ID strings in Vercel AI SDK.
Pricing on Vercel AI Gateway
| Type | Price (per 1M) |
|---|---|
| Input tokens | $0.72 |
| Output tokens | $0.72 |
Capabilities
About Llama 3.1 70B Instruct
The Llama 3.1 70B Instruct model is a cutting-edge large language model with 70 billion parameters, designed for instruction-following tasks. It features multilingual capabilities, supporting languages like English, German, French, and others. Fine-tuned using supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF), it excels in understanding and responding to user instructions. The model can handle a context length of up to 128k tokens, making it suitable for complex dialogue systems and applications requiring detailed responses. It outperforms many existing open-source and proprietary models on various industry benchmarks, making it ideal for conversational AI, content generation, and data synthesis tasks. For more details, visit the Hugging Face page [1].