Modal Labs

Integrate Modal with Portkey for seamless completions and streaming.

Quick Start

Get started with Modal in under 2 minutes:

Python

from portkey_ai import Portkey

# 1. Install: pip install portkey-ai
# 2. Add @modal provider in model catalog
# 3. Use it:

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@modal/your-model-name",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai'

// 1. Install: npm install portkey-ai
// 2. Add @modal provider in model catalog
// 3. Use it:

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY"
})

const response = await portkey.chat.completions.create({
    model: "@modal/your-model-name",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL

# 1. Install: pip install openai portkey-ai
# 2. Add @modal provider in model catalog
# 3. Use it:

client = OpenAI(
    api_key="PORTKEY_API_KEY",  # Portkey API key
    base_url=PORTKEY_GATEWAY_URL
)

response = client.chat.completions.create(
    model="@modal/your-model-name",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import OpenAI from "openai"
import { PORTKEY_GATEWAY_URL } from "portkey-ai"

// 1. Install: npm install openai portkey-ai
// 2. Add @modal provider in model catalog
// 3. Use it:

const client = new OpenAI({
    apiKey: "PORTKEY_API_KEY",  // Portkey API key
    baseURL: PORTKEY_GATEWAY_URL
})

const response = await client.chat.completions.create({
    model: "@modal/your-model-name",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

# 1. Add @modal provider in model catalog
# 2. Use it:

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -d '{
    "model": "@modal/your-model-name",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Tip: You can also set provider="@modal" in Portkey() and use just model="your-model-name" in the request.

Add Provider in Model Catalog

Before making requests, add Modal to your Model Catalog:

1. Go to Model Catalog → Add Provider
2. Select Modal
3. Enter your Modal API credentials
   • Modal supports API Key (Bearer token) or Custom Headers (modal-key and modal-secret)
4. Custom Host: Enter your Modal deployment endpoint (e.g., https://your-modal-deployment.modal.run)
5. Name your provider (e.g., modal)

Note: Custom host configuration is set once in Model Catalog and applies to all requests using this provider.

Complete Setup Guide

See all setup options and detailed configuration instructions

Modal Documentation

Explore the official Modal documentation

---

Modal Capabilities

Streaming

Stream responses for real-time output:

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY")

stream = portkey.chat.completions.create(
    model="@modal/your-model",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY'
});

const stream = await portkey.chat.completions.create({
    model: '@modal/your-model',
    messages: [{ role: 'user', content: 'Tell me a story' }],
    stream: true
});

for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

---

Supported Endpoints and Parameters

Endpoint	Supported Parameters
/chat/completions	messages, max_tokens, temperature, top_p, stream, presence_penalty, frequency_penalty

---

Next Steps

Gateway Configs

Add fallbacks, load balancing, and more

Observability

Monitor and trace your Modal requests

Prompt Library

Manage and version your prompts

Metadata

Add custom metadata to requests

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation