Predibase - Portkey Docs

Quick Start

Get started with Predibase in under 2 minutes:

from portkey_ai import Portkey

# 1. Install: pip install portkey-ai
# 2. Add @predibase provider in model catalog
# 3. Use it:

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@predibase/llama-3-8b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai'

// 1. Install: npm install portkey-ai
// 2. Add @predibase provider in model catalog
// 3. Use it:

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY"
})

const response = await portkey.chat.completions.create({
    model: "@predibase/llama-3-8b-instruct",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL

# 1. Install: pip install openai portkey-ai
# 2. Add @predibase provider in model catalog
# 3. Use it:

client = OpenAI(
    api_key="PORTKEY_API_KEY",  # Portkey API key
    base_url=PORTKEY_GATEWAY_URL
)

response = client.chat.completions.create(
    model="@predibase/llama-3-8b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import OpenAI from "openai"
import { PORTKEY_GATEWAY_URL } from "portkey-ai"

// 1. Install: npm install openai portkey-ai
// 2. Add @predibase provider in model catalog
// 3. Use it:

const client = new OpenAI({
    apiKey: "PORTKEY_API_KEY",  // Portkey API key
    baseURL: PORTKEY_GATEWAY_URL
})

const response = await client.chat.completions.create({
    model: "@predibase/llama-3-8b-instruct",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

# 1. Add @predibase provider in model catalog
# 2. Use it:

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -d '{
    "model": "@predibase/llama-3-8b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Add Provider in Model Catalog

Before making requests, add Predibase to your Model Catalog:

Go to Model Catalog → Add Provider
Select Predibase
Enter your Predibase API key
Name your provider (e.g., predibase)

Complete Setup Guide

See all setup options and detailed configuration instructions

Predibase Capabilities

Serverless Endpoints

Predibase offers LLMs like Llama 3, Mistral, Gemma, etc. on its serverless infrastructure that you can query instantly.

Sending Predibase Tenant IDPredibase expects your account tenant ID along with the API key in each request. With Portkey, you can send your Tenant ID with the user param while making your request.

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="llama-3-8b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    user="PREDIBASE_TENANT_ID"  # Required: Your Predibase tenant ID
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "llama-3-8b-instruct",
    messages: [{ role: "user", content: "Hello!" }],
    user: "PREDIBASE_TENANT_ID"  // Required: Your Predibase tenant ID
});

console.log(response.choices[0].message.content);

Using Fine-Tuned Models

Predibase allows you to deploy and use fine-tuned models with adapters. Use the special format with your model identifier from your Predibase dashboard:

Fine-Tuned Model Format:
model = base_model:adapter-repo-name/adapter-version-numberFor example: llama-3-8b:sentiment-analysis/1

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="llama-3-8b:sentiment-analysis/1",  # Base model + adapter
    messages=[{"role": "user", "content": "This product is amazing!"}],
    user="PREDIBASE_TENANT_ID"
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "llama-3-8b:sentiment-analysis/1",  // Base model + adapter
    messages: [{ role: "user", content: "This product is amazing!" }],
    user: "PREDIBASE_TENANT_ID"
});

console.log(response.choices[0].message.content);

Dedicated Deployments

Route requests to your dedicated deployed models by passing the deployment name in the model parameter:

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="my-dedicated-mistral-deployment",  # Your deployment name
    messages=[{"role": "user", "content": "Hello!"}],
    user="PREDIBASE_TENANT_ID"
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "my-dedicated-mistral-deployment",  // Your deployment name
    messages: [{ role: "user", content: "Hello!" }],
    user: "PREDIBASE_TENANT_ID"
});

console.log(response.choices[0].message.content);

JSON Schema Mode

Enforce JSON schema for all Predibase models by setting response_format to json_object with your schema:

from portkey_ai import Portkey
from pydantic import BaseModel, constr

# Define JSON Schema with Pydantic
class Character(BaseModel):
    name: constr(max_length=10)
    age: int
    strength: int

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="llama-3-8b",
    messages=[{"role": "user", "content": "Create a character profile"}],
    user="PREDIBASE_TENANT_ID",
    response_format={
        "type": "json_object",
        "schema": Character.schema()
    }
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "llama-3-8b",
    messages: [{ role: "user", content: "Create a character profile" }],
    user: "PREDIBASE_TENANT_ID",
    response_format: {
        type: "json_object",
        schema: {
            properties: {
                name: { maxLength: 10, title: "Name", type: "string" },
                age: { title: "Age", type: "integer" },
                strength: { title: "Strength", type: "integer" }
            },
            required: ["name", "age", "strength"],
            title: "Character",
            type: "object"
        }
    }
});

console.log(response.choices[0].message.content);

Supported Models

Predibase provides access to various open-source and fine-tuned models:

Llama 3 (various sizes)
Mistral
Zephyr
Your custom fine-tuned models

Check Predibase’s documentation for the complete model list and fine-tuning options.

Next Steps

Gateway Configs

Add fallbacks, load balancing, and more

Observability

Monitor and trace your Predibase requests

Prompt Library

Manage and version your prompts

Fine-Tuning Logs

Track performance of fine-tuned models

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation

​Quick Start

​Add Provider in Model Catalog

Complete Setup Guide

​Predibase Capabilities

​Serverless Endpoints

​Using Fine-Tuned Models

​Dedicated Deployments

​JSON Schema Mode

​Supported Models

​Next Steps

Gateway Configs

Observability

Prompt Library

Fine-Tuning Logs

SDK Reference

Quick Start

Add Provider in Model Catalog

Predibase Capabilities

Serverless Endpoints

Using Fine-Tuned Models

Dedicated Deployments

JSON Schema Mode

Supported Models

Next Steps