Model Routing

Quick Access: Connections | Model Collections | Settings

LangMart provides flexible ways to route your API requests to the right model. Understanding model routing helps you optimize for cost, latency, and access control.

Overview

When you send a request to LangMart, the system determines which model and connection to use based on the model name format you provide:

┌─────────────────────────────────────────────────────────────────────────┐
│                        Model Routing Flow                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   API Request                                                            │
│       │                                                                  │
│       ▼                                                                  │
│   ┌─────────────────────┐                                               │
│   │ Parse Model Name    │                                               │
│   └──────────┬──────────┘                                               │
│              │                                                           │
│    ┌─────────┼─────────────────────────────────┐                        │
│    │         │                │                │                        │
│    ▼         ▼                ▼                ▼                        │
│ provider/  handle/          collection/     auto(free)                  │
│ model      provider/model   name                                        │
│    │         │                │                │                        │
│    ▼         ▼                ▼                ▼                        │
│ Priority    Direct          Collection      System                      │
│ Routing     Access          Strategy        Collection                  │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

1. Direct Model Access

The simplest way to use a model is with the standard provider/model-name format. This routes to available connections based on priority.

Format

provider/model-name

Examples

Code Example

from openai import OpenAI

client = OpenAI(
    api_key="your-langmart-api-key",
    base_url="https://api.langmart.ai/v1"
)

# Use direct model access
response = client.chat.completions.create(
    model="groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}]
)

curl -X POST https://api.langmart.ai/v1/chat/completions \
  -H "Authorization: Bearer sk-langmart-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

---

2. Priority-Based Routing

When you have the same model available from multiple connections (e.g., your private Groq key and an organization-shared Groq key), the Priority Score determines which connection is used.

How Priority Works

Priority Resolution Example

Request: model="groq/llama-3.3-70b-versatile"

Available Connections:
┌────────────────────────────────────────────────────────────────┐
│  Priority 100  │  Your Private Groq Key      │  ← Used first  │
├────────────────────────────────────────────────────────────────┤
│  Priority 50   │  Organization Groq Key      │  ← Fallback    │
├────────────────────────────────────────────────────────────────┤
│  Priority 0    │  Platform Default           │  ← Last resort │
└────────────────────────────────────────────────────────────────┘

Customizing Priority

You can adjust connection priorities in the Connections page:

1. Navigate to Connections
2. Find the connection you want to adjust
3. Click Edit or Priority Settings
4. Set your preferred priority score (0-100)
5. Save changes

Use Cases

---

3. Direct Access with LangMart Handles

LangMart Handles allow you to access specific private or organization connections directly, bypassing priority routing. This is useful when you need to ensure a request uses a particular connection.

What is a LangMart Handle?

A LangMart Handle is a unique identifier that represents you or your organization on the platform. Every user and organization has a handle that can be used for direct model routing.

Finding Your LangMart Handle

• User Handle: Go to Settings > Account tab > "Your Handle" section
• Organization Handle: Go to Connections and view connection details, or check your organization settings

Handle-Based Model Format

langmart-handle/provider/model-name

Examples

Code Example

from openai import OpenAI

client = OpenAI(
    api_key="your-langmart-api-key",
    base_url="https://api.langmart.ai/v1"
)

# Use your private connection directly (user handle)
response = client.chat.completions.create(
    model="john-dev/groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Use organization connection directly (org handle)
response = client.chat.completions.create(
    model="acme-corp/openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

When to Use LangMart Handle-Based Routing

---

4. Model Collections

Model collections allow you to group multiple models together with a routing strategy. When you use a collection, the system automatically selects a model from the collection based on the strategy.

What is a Collection?

A collection is a named group of models with a routing strategy:

System Collections

LangMart provides built-in system collections:

Collection Format

collection/collection-name

Or for system collections:

auto(free)

Creating a Collection

1. Navigate to Model Collections
2. Click Create Collection
3. Name your collection
4. Add models to the collection
5. Choose a routing strategy
6. Save

Code Example

from openai import OpenAI

client = OpenAI(
    api_key="your-langmart-api-key",
    base_url="https://api.langmart.ai/v1"
)

# Use the default free collection
response = client.chat.completions.create(
    model="auto(free)",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Use a custom collection
response = client.chat.completions.create(
    model="collection/my-coding-models",
    messages=[{"role": "user", "content": "Write a Python function"}]
)

Example Collection: Coding Models

Collection: my-coding-models
Strategy: Fallback

Models (in order):
1. anthropic/claude-3-5-sonnet-20241022  ← Try first (best for code)
2. openai/gpt-4o                          ← Fallback
3. groq/llama-3.3-70b-versatile           ← Last resort (fast & free)

Collection Use Cases

---

Routing Decision Flowchart

┌─────────────────────────────────────────────────────────────────────────┐
│                     Model Name Parsing                                   │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   Model Name Format                   Action                             │
│   ─────────────────                   ──────                             │
│                                                                          │
│   auto(free)                    →     Use system free collection         │
│   auto(*)                       →     Use system collection              │
│                                                                          │
│   collection/name               →     Use custom collection strategy     │
│                                                                          │
│   langmart-handle/provider/model →    Direct access to specific          │
│                                       connection (user or org handle)    │
│                                                                          │
│   provider/model                →     Priority-based routing             │
│                                       (private → org → platform)         │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

---

Best Practices

1. Set Up Your LangMart Handles

Configure meaningful LangMart Handles for easy identification:

• User Handle: Your name or username (e.g., john, john-dev)
• Organization Handle: Company name or team (e.g., acme-corp, my-startup)

2. Organize Connection Priorities

3. Use Collections for Resilience

Create collections with fallback strategies to ensure high availability:

Primary: openai/gpt-4o
Fallback 1: anthropic/claude-3-5-sonnet-20241022
Fallback 2: groq/llama-3.3-70b-versatile

4. Monitor Routing Decisions

Check Request Logs to see which connection was used for each request:

• View the "Resolution" column for routing method
• Check "Connection" to see which API key was used
• Use filters to analyze routing patterns

---

Troubleshooting

Model Not Found

Wrong Connection Used

Access Denied

---

Quick Reference

---

Related Topics

• Connections Overview - Managing provider connections
• Adding Connections - Set up new connections
• Model Collections - Creating and managing collections
• Request Logs - Monitor routing decisions