Traces, prompts & budgets

Beyond cost and usage, the dashboard has five views. Traces, prompts and model comparison come from two optional request headers; budgets and anomaly alerts are computed for you.

Group calls into traces / sessions

Send an x-aicc-trace header with a shared id across the LLM calls that belong to one app request or agent run. The Traces view then lists each session and, on click, shows a timeline of its calls with per-call cost, tokens and latency.

import uuid
trace = str(uuid.uuid4())            # one id per user request / agent run

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={"x-aicc-trace": trace},   # OpenAI SDK: extra_headers
)

Any HTTP client works - it is just a header. x-aicc-session is accepted as an alias.

Track prompt versions

Send x-aicc-prompt (a template name) and optionally x-aicc-prompt-version. The Prompts view shows requests, error rate, average cost, tokens and latency per prompt version - so when you change a prompt you can see if cost or quality drifted.

# any language - just two headers
-H "x-aicc-prompt: claim-triage"
-H "x-aicc-prompt-version: v3"

Compare models

The Models view needs no headers. It derives, per model, the effective cost per million tokens, p50/p95 latency and error rate from real traffic, side by side - so a model swap is a measured decision.

Budgets & alerts

Set a monthly budget (USD) per project in the Alerts view, with an alert threshold percentage. The gateway tracks current-month spend and raises an alert when a project crosses the threshold or goes over budget. It also flags per-project error-rate and p95-latency breaches over the last 24h. Set an alertWebhook in config to receive a POST when new alerts fire.

// aicc.config.json
{
  "alertWebhook": "https://hooks.example.com/aicc"   // optional: POSTed newly-fired alerts
}

Budgets are admin-managed and stored in dataDir/budgets.json.

Anomaly detection

The Alerts view surfaces rule-based anomalies over the last 30 days: a project whose daily spend is more than ~2x its own median day (a cost spike), or a day with an unusually high error rate (an error burst). The rules are explainable - no ML, no external service - and each anomaly says exactly why it fired.

Provider routing (failover & load-balancing)

Define a route - a virtual provider that fans requests across an ordered pool of same-schema providers - and call it at /r/<route>/…. On a network error or a retryable status (429/500/502/503/504 by default), the gateway falls over to the next member before any response byte is streamed. It is opt-in, needs no client code change, and uses each member's central key. Every attempt is logged, so fallbacks show up in the request feed with a via <route> tag.

// aicc.config.json
{
  "keys": { "groq": "gsk_...", "together": "...", "openrouter": "..." },
  "routes": {
    "chat": {
      "members": ["groq", "together", "openrouter"],
      "strategy": "failover",          // or "round-robin"
      "retryOn": [429, 500, 502, 503, 504]
    }
  }
}

# point the base URL at the route instead of a single provider
client = OpenAI(base_url="http://localhost:4321/p/app/r/chat/v1")

Members should share a schema (all OpenAI-compatible, say) - the gateway forwards your body unchanged and does not translate between provider APIs.

Roles & per-project grants

Beyond admin and member, there is a read-only viewer role. Any non-admin user can also be granted access to specific projects (allowedProjects), which stacks on top of their team's projects - so you can give someone exactly the projects they need without a team. Manage roles and grants in the settings panel. Full OIDC/SAML SSO is on the roadmap; the built-in accounts cover single-team self-hosting today.