Providers & pricing

Built-in support for the major providers plus anything that speaks the OpenAI schema.

Built-in providers

Add a custom provider

Anything OpenAI-compatible (Azure OpenAI, vLLM, an internal gateway) works via config:

"providers": {
  "azure":  { "upstream": "https://myorg.openai.azure.com", "kind": "openai", "authHeader": "api-key" },
  "vllm":   { "upstream": "http://gpu-box:8000", "kind": "openai" }
}
// then call:  http://localhost:4321/p/my-app/azure/v1/chat/completions

Registered providers show up in the start banner (marked custom, with their upstream and where the API key comes from) and in npx ai-command-center snippets - if yours is missing there, the registration did not take (typo in the config, or the wrong config file picked up).

Custom providers need pricing overrides. Cost is looked up on the live LiteLLM price sheet plus the shipped table - both only know mainstream cloud models. Models served by a custom provider (an internal fine-tune, a regional provider's *-instruct models, every Azure deployment alias) are usually absent, so their requests are recorded with full token counts but unpriced (cost 0) until a pricing block covers them. Pair every custom provider with its prices:

"providers": {
  "sarvam": { "upstream": "https://api.sarvam.ai", "kind": "openai", "keyEnv": "SARVAM_API_KEY" }
},
"pricing": {
  "sarvam-m": { "in": 0.8, "out": 1.6 },   // USD per 1M tokens
  "saarika":  { "in": 0.5, "out": 0.5 }
}

A provider-wide wildcard like "sarvam:*": { "in": 0, "out": 0 } is a deliberate policy, not a fallback: once set, only provider-qualified entries ("sarvam:sarvam-m": …) override it - plain model keys for that provider are ignored. Use it to price a whole provider at zero (e.g. self-hosted), qualified keys plus the wildcard if you want per-model prices with a default underneath.

npx ai-command-center stats tells you when this is needed - it prints the unpriced request count and the exact model names to add overrides for.

How streaming usage is captured

Streaming responses are passed through byte-for-byte while usage is parsed on the side. For OpenAI-style streams the gateway quietly adds stream_options: {include_usage: true}so the final chunk carries token counts - and then strips that extra usage-only chunk back out, so clients that never asked for it still see a normal stream.

Pricing

Cost is computed from real token counts against a shipped price table (pricing/pricing.json, USD per 1M tokens, longest-prefix match on the model name). Prices drift - the table ships as a sane default; verify against provider price pages and override in config:

"pricing": {
  "gpt-4o-mini": { "in": 0.15, "out": 0.6 },     // override a shipped price
  "my-finetune": { "in": 1.0, "out": 4.0 },       // add a new model
  "openrouter:*": { "in": 0, "out": 0 }           // provider-wide default
}

Requests on a model with no price are logged with full token counts and flagged as unpriced (never guessed). The eval report checks cost math is exact for the priced models.