Llm Finops Router

LLM FinOps & Dynamic Model Router

English | Bahasa Indonesia

English

Orchestration & Integration

• Upstream & Orchestration: zero-to-prod-orchestrator (Phase 4 AI integration), brainstorming, session-memory-manager.
• Model Routing & Cascading: adaptive-model-cascade (complexity-scored model selection), thinking-budget-orchestrator (reasoning token envelopes), test-time-compute-optimizer (compute cost scaling).
• Prompt Caching & Context: kv-cache-prefix-optimizer (KV-cache hit maximization), context-window-engineer (per-zone token cost accounting).
• Observability & Infrastructure: llm-observability-expert (Langfuse traces + cost dashboards), cloud-hosting-expert (gateway deployment on Vercel, Cloudflare, AWS), vercel-ai-sdk-expert (edge streaming & tool-call routing), multi-agent-orchestration (per-agent budget envelopes).

Purpose

Optimize operational costs, reliability, and latency of multi-model LLM architectures using dynamic routing, provider fallback chains, semantic caching, and real-time FinOps analytics. Target: 40–70% reduction in monthly LLM spend without degrading output quality.

1. LiteLLM v1.x — Deep Configuration Guide

LiteLLM provides a unified OpenAI-compatible interface across 100+ LLM providers with built-in fallback, retries, and budget management.

Install & Start Proxy

bashCopy code
pip install litellm[proxy]
litellm --config litellm_config.yaml --port 4000

litellm_config.yaml — Full Production Config

yamlCopy code
model_list:
  - model_name: high-fast-tier       # Virtual name used by clients
    litellm_params:
      model: xai/grok-4.7-high-fast
      api_key: os.environ/XAI_API_KEY
      rpm: 1200
      tpm: 3000000

  - model_name: fast-tier            # Rapid AST diff and code edits
    litellm_params:
      model: cursor/composer-2.5-fast
      api_key: os.environ/CURSOR_API_KEY
      rpm: 1000

  - model_name: medium-tier          # Balanced architecture & strict schema contracts
    litellm_params:
      model: gemini/gemini-4-argon
      api_key: os.environ/GEMINI_API_KEY
      rpm: 300
      tpm: 2000000

  - model_name: reasoning-tier       # High-complexity fullstack craftsmanship
    litellm_params:
      model: anthropic/claude-sonnet-5.5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 100
      tpm: 500000

  - model_name: reasoning-tier       # Fallback 1 for reasoning-tier
    litellm_params:
      model: gemini/gemini-4-pro
      api_key: os.environ/GEMINI_API_KEY
      rpm: 60

  - model_name: embedding-tier
    litellm_params:
      model: text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: least-busy        # Options: simple-shuffle | least-busy | latency-based-routing | cost-based-routing
  enable_pre_call_checks: true
  num_retries: 3
  retry_after: 2                      # seconds between retries
  timeout: 60
  fallbacks:
    - high-fast-tier: [xai/grok-4.6-high-fast, cursor/composer-2.5-fast]
    - medium-tier: [openai/gpt-5.6-sol, anthropic/claude-opus-5.5, anthropic/claude-opus-4.6, gemini/gemini-3.8-flash]
    - reasoning-tier: [gemini/gemini-4-pro, gemini/gemini-3.8-flash, anthropic/claude-fable-5.1, anthropic/claude-opus-4.6, muse/muse-wonder-high]
  context_window_fallbacks:
    - reasoning-tier: [gemini/gemini-4-argon, gemini/gemini-4-pro, gemini/gemini-3.8-flash]  # 1M+ to 10M+ context fallback

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: 6379
    password: os.environ/REDIS_PASSWORD
    ttl: 600                          # Cache TTL in seconds
    similarity_threshold: 0.92        # Semantic cache hit threshold

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL  # PostgreSQL for spend tracking

# Virtual Keys with per-user budget limits
virtual_keys:
  - key: sk-agent-research
    user_id: agent-research
    max_budget: 5.00                  # USD per day
    budget_duration: 1d
    allowed_models: [fast-tier, embedding-tier]
  - key: sk-agent-code
    user_id: agent-code
    max_budget: 10.00
    budget_duration: 1d
    allowed_models: [fast-tier, reasoning-tier]

Python: Dynamic Routing via LiteLLM Client

pythonCopy code
import litellm
from litellm import acompletion

async def route_by_complexity(prompt: str, complexity_score: float) -> str:
    """
    Routes to fast-tier (<0.5) or reasoning-tier (>=0.5) based on score.
    complexity_score: 0.0 (trivial) → 1.0 (very complex)
    """
    model = "reasoning-tier" if complexity_score >= 0.5 else "fast-tier"

    response = await acompletion(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        api_base="http://localhost:4000",
        api_key="sk-agent-code",
        metadata={
            "generation_name": "dynamic-route",
            "tags": ["finops", f"complexity:{complexity_score:.2f}"],
        },
    )
    return response.choices[0].message.content

2. Portkey AI Gateway — Production Setup

Portkey adds a middleware layer over LLM providers with semantic caching, canary routing, observability, and guardrails.

bashCopy code
npm install portkey-ai

TypeScript: Portkey Gateway with Fallback & Semantic Cache

typescriptCopy code
import Portkey from "portkey-ai";

const portkey = new Portkey({
  apiKey: process.env.PORTKEY_API_KEY!,
  config: {
    strategy: { mode: "fallback" },
    targets: [
      {
        provider: "anthropic",
        api_key: process.env.ANTHROPIC_API_KEY!,
        override_params: { model: "claude-opus-4-5", max_tokens: 4096 },
        weight: 1,
      },
      {
        // Fallback if Anthropic fails or is rate-limited
        provider: "google",
        api_key: process.env.GEMINI_API_KEY!,
        override_params: { model: "gemini-4-argon-medium", max_tokens: 4096 },
      },
    ],
    cache: {
      mode: "semantic",               // Semantic similarity matching
      max_age: 3600,                  // 1 hour TTL
    },
    retry: { attempts: 3, on_status_codes: [429, 500, 502, 503] },
  },
});

export async function callWithPortkey(
  prompt: string,
  userId: string,
): Promise<string> {
  const response = await portkey.chat.completions.create({
    messages: [{ role: "user", content: prompt }],
    model: "claude-opus-4-5",
    // Portkey metadata for observability
    // @ts-expect-error — Portkey extends OpenAI types
    metadata: { _user: userId, environment: process.env.NODE_ENV },
  });

  return response.choices[0].message.content ?? "";
}

Portkey Canary Routing (A/B Test New Models)

typescriptCopy code
const portkey = new Portkey({
  apiKey: process.env.PORTKEY_API_KEY!,
  config: {
    strategy: { mode: "loadbalance" },
    targets: [
      {
        provider: "anthropic",
        api_key: process.env.ANTHROPIC_API_KEY!,
        override_params: { model: "claude-opus-4-5" },
        weight: 80,                   // 80% traffic to stable model
      },
      {
        provider: "anthropic",
        api_key: process.env.ANTHROPIC_API_KEY!,
        override_params: { model: "claude-opus-4-6-preview" },  // new model canary
        weight: 20,                   // 20% canary traffic
      },
    ],
  },
});

3. RouteLLM (Stanford) — Complexity Classifier Router

RouteLLM trains a binary classifier to decide whether a query needs a strong model or a weak model, achieving strong-model call reduction of 40-85% while preserving quality.

bashCopy code
pip install routellm

Setup & Router Configuration

pythonCopy code
from routellm.controller import Controller

# Initialize with a pre-trained classifier (matrix-factorization or causal-llm)
controller = Controller(
    routers=["mf"],                   # 'mf' = matrix-factorization (fastest)
    strong_model="claude-sonnet-5.5",
    weak_model="grok-4.7-high-fast",
    config={
        "mf": {
            "checkpoint_path": "routellm/mf-gpt4-turbo",  # HF checkpoint
        }
    },
    api_base=None,                    # Uses provider API directly
    api_key=None,
)

# Threshold: higher = use weak model more aggressively
# 0.11875 = calibrated to 50% strong model call rate on MMLU benchmark
response = controller.chat.completions.create(
    model="router-mf-0.11875",        # router-{name}-{threshold}
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(response.choices[0].message.content)

Custom Task-Based Routing

pythonCopy code
async def intelligent_route(task_type: str, prompt: str) -> str:
    """Route based on known task taxonomy (late-2026 frontier tiers)."""
    FAST_TASKS = {"classification", "extraction", "summarization", "translation", "ast_scan"}
    SLOW_TASKS = {"code_generation", "reasoning", "math", "planning", "security_audit"}

    use_strong = task_type in SLOW_TASKS
    model = "claude-sonnet-5.5" if use_strong else "grok-4.7-high-fast"

    import openai
    client = openai.AsyncOpenAI()
    resp = await client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

4. Cost Calculation & Budget Enforcement

Per-Model Pricing Table (Late-2026 Frontier Models, USD per 1M tokens)

TypeScript: Real-Time Cost Calculator

typescriptCopy code
interface ModelPricing {
  inputPerMillion: number;
  outputPerMillion: number;
  cachedInputPerMillion?: number;
}

const MODEL_PRICING: Record<string, ModelPricing> = {
  "grok-4.7-high-fast":      { inputPerMillion: 0.50,  outputPerMillion: 2.00,  cachedInputPerMillion: 0.10 },
  "grok-4.6-high-fast":      { inputPerMillion: 0.40,  outputPerMillion: 1.60,  cachedInputPerMillion: 0.08 },
  "composer-2.5-fast":       { inputPerMillion: 0.30,  outputPerMillion: 1.20,  cachedInputPerMillion: 0.05 },
  "claude-opus-5.5":         { inputPerMillion: 10.00, outputPerMillion: 50.00, cachedInputPerMillion: 1.00 },
  "claude-opus-4.6":         { inputPerMillion: 8.00,  outputPerMillion: 40.00, cachedInputPerMillion: 0.80 },
  "gemini-4-argon":          { inputPerMillion: 0.80,  outputPerMillion: 3.20,  cachedInputPerMillion: 0.15 },
  "gemini-3.8-flash":        { inputPerMillion: 0.10,  outputPerMillion: 0.40,  cachedInputPerMillion: 0.025 },
  "gpt-5.6-sol":             { inputPerMillion: 1.50,  outputPerMillion: 6.00,  cachedInputPerMillion: 0.35 },
  "claude-sonnet-5.5":       { inputPerMillion: 3.00,  outputPerMillion: 15.00, cachedInputPerMillion: 0.30 },
  "claude-fable-5.1":        { inputPerMillion: 2.50,  outputPerMillion: 12.50, cachedInputPerMillion: 0.25 },
  "muse-wonder-high":        { inputPerMillion: 3.50,  outputPerMillion: 17.50, cachedInputPerMillion: 0.35 },
  "gemini-4-pro":            { inputPerMillion: 1.25,  outputPerMillion: 10.00, cachedInputPerMillion: 0.31 },
};

export function calculateCost(
  model: string,
  inputTokens: number,
  outputTokens: number,
  cachedInputTokens = 0,
): number {
  const pricing = MODEL_PRICING[model];
  if (!pricing) throw new Error(`Unknown model: ${model}`);

  const inputCost = (inputTokens / 1_000_000) * pricing.inputPerMillion;
  const outputCost = (outputTokens / 1_000_000) * pricing.outputPerMillion;
  const cachedCost = pricing.cachedInputPerMillion
    ? (cachedInputTokens / 1_000_000) * pricing.cachedInputPerMillion
    : 0;

  return inputCost + outputCost + cachedCost;
}

// Token Budget Guard — throws if projected cost exceeds daily cap
export async function guardedLLMCall(
  model: string,
  messages: Array<{ role: string; content: string }>,
  dailyBudgetUSD: number,
  spentTodayUSD: number,
): Promise<string> {
  const estimatedInputTokens = messages.reduce(
    (sum, m) => sum + Math.ceil(m.content.length / 4),
    0,
  );
  const estimatedCost = calculateCost(model, estimatedInputTokens, estimatedInputTokens * 2);

  if (spentTodayUSD + estimatedCost > dailyBudgetUSD) {
    // Availability Fallback Invariant: Cascade to ultra-efficient Gemini 3.8 Flash
    const cheaperModel = "gemini-3.8-flash";
    console.warn(`Budget cap approaching. Cascading to ${cheaperModel}.`);
    return guardedLLMCall(cheaperModel, messages, dailyBudgetUSD, spentTodayUSD);
  }

  const { OpenAI } = await import("openai");
  const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
  const resp = await client.chat.completions.create({ model, messages } as Parameters<typeof client.chat.completions.create>[0]);
  return resp.choices[0].message.content ?? "";
}

5. Monitoring, Alerts & Daily Cost Caps

Langfuse Integration (spend tracking + tracing)

pythonCopy code
from langfuse import Langfuse
from langfuse.decorators import langfuse_context, observe

langfuse = Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com"),
)

@observe(name="llm-call", as_type="generation")
async def tracked_llm_call(model: str, prompt: str, user_id: str) -> str:
    import litellm
    resp = await litellm.acompletion(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        metadata={"langfuse_user_id": user_id},
    )

    # Record cost for this generation
    cost = litellm.completion_cost(completion_response=resp)
    langfuse_context.update_current_observation(
        usage={
            "input": resp.usage.prompt_tokens,
            "output": resp.usage.completion_tokens,
            "unit": "TOKENS",
            "total_cost": cost,
        }
    )
    return resp.choices[0].message.content

Daily Cost Cap via Redis Counter

typescriptCopy code
import { Redis } from "@upstash/redis";

const redis = new Redis({
  url: process.env.UPSTASH_REDIS_REST_URL!,
  token: process.env.UPSTASH_REDIS_REST_TOKEN!,
});

const DAILY_BUDGET_USD = parseFloat(process.env.LLM_DAILY_BUDGET_USD ?? "50");
const ALERT_THRESHOLD = 0.85; // Alert at 85% of budget

export async function trackAndCheckBudget(
  costUSD: number,
  slackWebhook?: string,
): Promise<{ allowed: boolean; spent: number; remaining: number }> {
  const key = `llm:spend:${new Date().toISOString().slice(0, 10)}`; // YYYY-MM-DD
  const spent = await redis.incrbyfloat(key, costUSD);
  await redis.expire(key, 86400 * 2); // 2 days TTL

  const remaining = DAILY_BUDGET_USD - spent;

  if (spent >= DAILY_BUDGET_USD * ALERT_THRESHOLD && slackWebhook) {
    await fetch(slackWebhook, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        text: `⚠️ *LLM Budget Alert*: \$${spent.toFixed(4)} spent of \$${DAILY_BUDGET_USD} daily budget (${((spent / DAILY_BUDGET_USD) * 100).toFixed(1)}%).`,
      }),
    });
  }

  return { allowed: spent <= DAILY_BUDGET_USD, spent, remaining };
}

6. Complexity Scorer — Routing Decision Engine

typescriptCopy code
export interface ComplexityScore {
  score: number;           // 0.0 → 1.0
  tier: "fast" | "reasoning";
  reason: string;
}

const HIGH_COMPLEXITY_PATTERNS = [
  /step[ -]by[ -]step/i,
  /reason(ing|ed)/i,
  /code\s+(generation|review|refactor)/i,
  /math|calculation|equation/i,
  /analyze|synthesize|compare/i,
  /plan|architect|design system/i,
];

const LOW_COMPLEXITY_PATTERNS = [
  /translate/i,
  /summarize|tldr/i,
  /classify|label/i,
  /extract\s+(name|date|email|phone)/i,
  /yes or no/i,
  /what is the (capital|meaning)/i,
];

export function scoreComplexity(prompt: string): ComplexityScore {
  let score = 0.3; // baseline

  for (const pattern of HIGH_COMPLEXITY_PATTERNS) {
    if (pattern.test(prompt)) score = Math.min(score + 0.2, 1.0);
  }
  for (const pattern of LOW_COMPLEXITY_PATTERNS) {
    if (pattern.test(prompt)) score = Math.max(score - 0.2, 0.0);
  }

  // Penalize long prompts slightly toward reasoning tier
  if (prompt.length > 2000) score = Math.min(score + 0.1, 1.0);

  const tier = score >= 0.5 ? "reasoning" : "fast";
  const reason =
    tier === "reasoning"
      ? `High complexity (score: ${score.toFixed(2)}) — using frontier model`
      : `Low complexity (score: ${score.toFixed(2)}) — using fast model`;

  return { score, tier, reason };
}

export function selectModel(tier: "fast" | "reasoning"): string {
  // Availability Fallback Invariant:
  // - reasoning: Claude Opus 5.5 -> fallback Claude Opus 4.6
  // - fast: Gemini 4 Argon -> fallback Gemini 3.8 Flash
  if (tier === "reasoning") {
    return process.env.FRONTIER_MODEL_AVAILABLE === "false" ? "claude-opus-4.6" : "claude-opus-5.5";
  }
  return process.env.FRONTIER_MODEL_AVAILABLE === "false" ? "gemini-3.8-flash" : "gemini-4-argon";
}

7. Full Routing Pipeline (TypeScript)

typescriptCopy code
import OpenAI from "openai";
import { scoreComplexity, selectModel } from "./complexity-scorer";
import { trackAndCheckBudget } from "./budget-tracker";
import { calculateCost } from "./cost-calculator";

const client = new OpenAI({
  baseURL: "http://localhost:4000",   // LiteLLM proxy
  apiKey: process.env.LITELLM_MASTER_KEY,
});

export async function smartRoute(
  prompt: string,
  userId: string,
): Promise<{ content: string; model: string; costUSD: number }> {
  const complexity = scoreComplexity(prompt);
  const model = selectModel(complexity.tier);

  const { allowed, spent } = await trackAndCheckBudget(0, process.env.SLACK_WEBHOOK);
  if (!allowed) {
    throw new Error(`Daily LLM budget exhausted. Spent: \$${spent.toFixed(2)}`);
  }

  const response = await client.chat.completions.create({
    model,
    messages: [{ role: "user", content: prompt }],
    user: userId,
  });

  const inputTokens = response.usage?.prompt_tokens ?? 0;
  const outputTokens = response.usage?.completion_tokens ?? 0;
  const costUSD = calculateCost(model, inputTokens, outputTokens);

  await trackAndCheckBudget(costUSD, process.env.SLACK_WEBHOOK);

  return {
    content: response.choices[0].message.content ?? "",
    model,
    costUSD,
  };
}

Sovereign Execution, Token Efficiency & Invariants

Bahasa Indonesia

Integrasi Orkestrasi

• Hulu & Orkestrasi: zero-to-prod-orchestrator (Integrasi AI Fase 4), brainstorming, session-memory-manager.
• Routing Model & Kaskade: adaptive-model-cascade (pemilihan model berbasis skor kompleksitas), thinking-budget-orchestrator (envelope token penalaran), test-time-compute-optimizer (skala biaya komputasi).
• Caching Prompt & Konteks: kv-cache-prefix-optimizer (maksimasi hit KV-cache), context-window-engineer (akuntansi biaya token per zona).
• Observabilitas & Infrastruktur: llm-observability-expert (trace Langfuse + dashboard biaya), cloud-hosting-expert (deployment gateway di Vercel, Cloudflare, AWS), vercel-ai-sdk-expert (routing streaming & tool-call di edge), multi-agent-orchestration (anggaran per-agen).

Tujuan

Mengoptimalkan biaya operasional, keandalan, dan latensi arsitektur LLM multi-model menggunakan routing dinamis, rantai fallback penyedia, semantic caching, dan analitik FinOps real-time. Target: pengurangan pengeluaran LLM bulanan 40–70% tanpa menurunkan kualitas output.

1. LiteLLM v1.x — Panduan Konfigurasi Mendalam

LiteLLM menyediakan antarmuka standar kompatibel OpenAI untuk 100+ penyedia LLM dengan fallback bawaan, retry, dan manajemen anggaran.

File konfigurasi utama: litellm_config.yaml

Konsep kunci:

• model_name (virtual name): nama yang dipakai klien — bisa didefinisikan lebih dari sekali untuk fallback otomatis
• litellm_params.model: nama model nyata di penyedia
• router_settings.routing_strategy: least-busy, latency-based-routing, atau cost-based-routing
• Virtual Keys: kunci API virtual dengan batas anggaran per pengguna (max_budget: 5.00 per hari)
• Semantic Cache: Redis-backed, similarity threshold 0.92 — kueri serupa dikembalikan dari cache

Lihat konfigurasi lengkap di Seksi English.

2. Portkey AI Gateway

Portkey bertindak sebagai middleware di atas penyedia LLM, memberikan:

• Semantic Caching: mengembalikan respons cached untuk kueri dengan kemiripan >92%, hemat 30–50% token
• Fallback Strategy: jika Anthropic gagal, otomatis turun ke Gemini
• Load Balance / Canary: kirim 80% traffic ke model stabil, 20% ke model baru untuk A/B test
• Retry Logic: retry otomatis pada status 429/500/502/503 hingga 3 kali
• Observability Hooks: setiap panggilan dicatat dengan metadata pengguna dan lingkungan

3. RouteLLM (Stanford)

RouteLLM melatih classifier biner untuk memutuskan apakah kueri memerlukan model kuat atau model lemah. Hasil benchmark: pengurangan panggilan model kuat 40–85% sambil mempertahankan kualitas.

• Router mf (matrix-factorization): tercepat, direkomendasikan untuk produksi
• Threshold 0.11875: dikalibrasi untuk 50% tingkat panggilan model kuat pada MMLU benchmark
• Gunakan format router-mf-{threshold} sebagai nama model

4. Formula Biaya & Penegakan Anggaran

Formula dasar:

Copy code
biaya = (input_token / 1_000_000 × harga_input)
      + (output_token / 1_000_000 × harga_output)
      + (cached_input_token / 1_000_000 × harga_cached)

Strategi penegakan anggaran:

1. Estimasi biaya sebelum memanggil LLM berdasarkan panjang prompt
2. Periksa saldo anggaran harian dari Redis
3. Jika saldo hampir habis → cascade ke model lebih murah, BUKAN tolak permintaan
4. Kirim alert Slack ketika pengeluaran mencapai 85% dari batas harian

5. Monitoring & Alert Biaya

Langfuse digunakan untuk:

• Melacak biaya per generasi (total_cost dalam metadata observasi)
• Dashboard penggunaan token per pengguna, per model
• Deteksi anomali pengeluaran (lonjakan tak terduga)

Redis Cost Counter (via Upstash):

• Key format: llm:spend:YYYY-MM-DD
• Increment atomic menggunakan INCRBYFLOAT
• Alert Slack otomatis saat 85% dari LLM_DAILY_BUDGET_USD terlampaui

6. Scorer Kompleksitas

Sistem scoring berbasis pola regex:

• Pattern complexity tinggi: reasoning, code generation, analyze, plan, math → tambah +0.2 skor
• Pattern complexity rendah: translate, summarize, classify, extract → kurangi -0.2 skor
• Prompt panjang >2000 karakter: tambah +0.1 skor
• Threshold routing: score ≥ 0.5 → reasoning tier; score < 0.5 → fast tier

Contoh implementasi TypeScript dan Python yang lengkap tersedia di seksi English.

Anti-Slop Checklist FinOps

• Tidak ada penolakan hard saat anggaran penuh — selalu cascade ke model lebih murah
• Setiap panggilan LLM direkam dengan user_id, model, cost_usd, input_tokens, output_tokens
• Semantic cache aktif di gateway (Portkey atau LiteLLM Redis)
• Alert Slack dikonfigurasi pada 85% batas harian
• Virtual keys digunakan untuk isolasi anggaran per agen/tim
• Complexity scorer diuji pada setidaknya 50 prompt representatif sebelum deploy

Matriks Eksekusi Berdaulat, Efisiensi Token & Invarian