Provider Mesh

The provider mesh is the routing layer between the web app / harness and the upstream model APIs. It owns: the model registry presets (ProviderConfig), per-request routing with automatic failover, circuit-breaker health tracking, and per-1K-token cost accounting.

Presets: model_harness/core/config.py:251-520
Mesh orchestrator: model_harness/mesh/mesh.py:73 (ProviderMesh)
One OpenAIAdapter per model, cached by model id (mesh/mesh.py:617-624)

Registered models

Ten presets are registered in DEFAULT_PROVIDERS (model_harness/core/config.py:509-520). DEFAULT_PROVIDER is KIMI_K27_CODE (config.py:507) and HarnessConfig.default_model is "kimi-for-coding" (config.py:616).

model_id	Provider key	Endpoint	API key env var	Ctx	Vision	Thinking / effort	$/1K in	$/1K out
`kimi-for-coding`	kimi	api.kimi.com/coding/v1	`KIMI_API_KEY` (fb `MOONSHOT_API_KEY`)	256K	no	thinking toggle; temp locked 1.0 / 0.6-off	0.001	0.002
`kimi-for-coding-highspeed`	kimi	api.kimi.com/coding/v1	`KIMI_API_KEY` (fb `MOONSHOT_API_KEY`)	256K	no	thinking toggle; temp locked 1.0 / 0.6-off	0.0005	0.001
`k3`	kimi	api.kimi.com/coding/v1	`KIMI_API_KEY` (fb `MOONSHOT_API_KEY`)	1M	yes	always reasons; effort low/high/max	0.003	0.015
`k3-256k`	kimi	api.kimi.com/coding/v1	`KIMI_API_KEY` (fb `MOONSHOT_API_KEY`)	256K	yes	effort low/high/max	0.0015	0.0075
`deepseek-v4-pro`	deepseek	Bailian token-plan endpoint	`BAILIAN_CODING_PLAN_API_KEY`	128K	yes	effort low/medium/high/max	0.004	0.012
`deepseek-v4-flash`	deepseek	api.deepseek.com (direct)	`DEEPSEEK_API_KEY`	128K	yes	effort low/medium/high/max	0.002	0.006
`glm-5.2`	glm	Bailian token-plan endpoint	`BAILIAN_CODING_PLAN_API_KEY` (no fallback)	128K	yes	thinking toggle + effort high/max	0.003	0.008
`qwen3.7-max`	qwen	Bailian token-plan endpoint	`BAILIAN_CODING_PLAN_API_KEY`	128K	yes	none	0.003	0.008
`qwen3.8-max`	qwen	Bailian token-plan endpoint	`BAILIAN_CODING_PLAN_API_KEY`	256K	yes	thinking toggle (default ON) + effort low/medium/high/max	0.003	0.008
`MiniMax-M3`	minimax	api.minimax.io/v1 (direct)	`MINIMAX_API_KEY` + `MINIMAX_GROUP_ID` header	128K	yes	none (`reasoning_split` extra_body)	0.002	0.006

Endpoints: Bailian Coding Plan vs direct

Four models are served through the Bailian Coding Plan (token plan) endpoint https://token-plan.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1 with BAILIAN_CODING_PLAN_API_KEY: deepseek-v4-pro (config.py:366-368), glm-5.2 (config.py:415-419), qwen3.7-max (config.py:463-466), qwen3.8-max (config.py:482-485). glm-5.2 deliberately has no ZAI_API_KEY fallback — a Zhipu key would be sent to the Alibaba endpoint and fail auth (config.py:416-417).

The rest hit provider-direct endpoints: the four Kimi models use api.kimi.com/coding/v1, deepseek-v4-flash uses api.deepseek.com, and MiniMax-M3 uses api.minimax.io/v1. PROVIDER_BASE_URLS (config.py:24-30) supplies default base URLs per provider key; an unset api_base on an unknown provider falls back to api.openai.com/v1 (config.py:153-157).

Key resolution and extra headers

ProviderConfig.get_api_key() resolves in order: direct api_key field → api_key_env → api_key_env_fallback (config.py:173-183). Models without a resolvable key are skipped by the UI/routing (has_api_key, config.py:185-187).

extra_headers values starting with $ are resolved from environment variables at call time and silently dropped when unset (config.py:220-240). MiniMax uses this for its required GroupId header (config.py:453-455). extra_body keys are merged into every request (e.g. MiniMax M3's {"reasoning_split": True}, config.py:456-457).

Request flow and failover

ProviderMesh.query() (mesh/mesh.py:252-404) streams text chunks. Steps:

Direct bypass — an explicit model_id with no strategy skips routing (mesh.py:296-312) but still gets capability validation and profile detection.
Profile detection — _detect_profile (mesh.py:648-683) estimates tokens (tiktoken cl100k via memory.dtos.estimate_tokens, ~4 chars/token fallback), classifies complexity simple/medium/complex, and flags tool use from keywords or a tools kwarg.
Capability pre-flight — candidates are filtered by supports_vision, supports_reasoning_effort, supports_thinking_toggle before any tokens are spent (mesh.py:329-359); the same guards run per call in _validate_capability_request (mesh.py:580-615) and again in the adapter (mesh/adapter.py:52-74).
Health filter — models whose circuit breaker rejects traffic are dropped; if all are unhealthy the full list is tried anyway (cold start, mesh.py:367-375).
Routing — the configured strategy returns a RoutingDecision with selected_model + alternatives; the ordered list is the fallback chain (mesh.py:377-393).

Registered strategies (mesh.py:129-139): first_available, round_robin, least_cost, least_latency, capability_match (the default, mesh/dtos.py:115), weighted, a_b_test.

Retry and fallback semantics

_execute_with_fallback (mesh.py:448-564):

Each model gets max_retries attempts (default 3, dtos.py:116).
Between attempts: exponential backoff retry_delay_ms * 2**attempt (default base 500 ms) when exponential_backoff is on (mesh.py:547-555).
After a model's retries are exhausted its failure is recorded on the circuit breaker and the next model in the chain is tried (mesh.py:557-559).
If every model fails: AllProvidersExhaustedError with code ALL_PROVIDERS_EXHAUSTED, listing models tried and errors (mesh.py:55-70, raised at mesh.py:562-564).
On success: latency is recorded for the least_latency strategy and the breaker records a success (mesh.py:486-488).

flowchart LR
    A[Client query] --> B[ProviderMesh.query]
    B --> C{model_id given}
    C -- yes --> D[Direct bypass with capability check]
    C -- no --> E[Detect QueryProfile]
    E --> F[Filter candidates by capability]
    F --> G[Drop circuit-broken models]
    G --> H[Strategy picks selected plus alternatives]
    D --> I[Execute with fallback]
    H --> I
    I --> J{Stream OK}
    J -- yes --> K[Record latency, cost, breaker success]
    J -- no --> L[Retry same model, exp backoff, max 3]
    L -- exhausted --> M[Breaker failure, next model in chain]
    M --> I
    M -- none left --> N[AllProvidersExhaustedError]

Circuit breaker and health

Per-model CircuitBreaker (mesh/health.py:27-133), defaults from MeshConfig (mesh/dtos.py:119-121):

CLOSED → OPEN after 5 consecutive failures.
OPEN → HALF_OPEN after a 30 s recovery timeout.
HALF_OPEN → CLOSED after 3 successful probe requests; any failure in HALF_OPEN reopens.

HealthChecker.health_status() maps breaker state to healthy / degraded / down per model (health.py:202-216). Proactive background pings are off by default (enable_proactive_health_checks=False, dtos.py:129) because they burn API quota; health is tracked reactively from real query traffic (mesh.py:210-226).

Cost accounting

CostTracker (mesh/cost_tracker.py:25) is in-memory. After each successful call the mesh records usage (mesh.py:510-523):

cost = (prompt_tokens / 1000) * cost_per_1k_input
     + (completion_tokens / 1000) * cost_per_1k_output   # rounded to 6 dp

(cost_tracker.py:73-75.) In the streaming path, token counts are estimates — prompt tokens from the QueryProfile estimate, completion tokens from estimate_tokens on the assembled output (mesh.py:490-494). Each UsageRecord (mesh/dtos.py:73-106) also carries prompt-cache counters: cached_tokens (Kimi/Moonshot shape) and prompt_cache_hit_tokens / prompt_cache_miss_tokens (DeepSeek shape), read from the adapter's last usage payload (mesh.py:496-508).

Queryable aggregates:

get_session_cost(session_id) — cumulative per-session cost (cost_tracker.py:156-161).
daily_cost / monthly_cost — per-model or aggregate (cost_tracker.py:102-134).
get_cost_summary() — daily/monthly totals, per-model breakdown, record counts, and a cache hit-rate summary (cost_tracker.py:172-208).

Budgets: per-model daily budgets and a monthly hard cap (default 500.0 USD, dtos.py:131-132) are configured via MeshConfig and applied to the tracker at mesh construction (mesh.py:106-109). within_budget / within_monthly_cap are check methods (cost_tracker.py:136-154); the tracker itself only records — enforcement is the caller's responsibility.

Thinking and reasoning_effort support

Two distinct mechanisms, both sent via extra_body (the OpenAI SDK rejects them as top-level kwargs, mesh/adapter.py:164-175):

Thinking toggle (supports_thinking_toggle) → extra_body["thinking"] = {"type": "enabled"|"disabled"}. Supported by: both kimi-for-coding variants, glm-5.2, qwen3.8-max (GUI default ON for qwen3.8 via thinking_default, config.py:498).
reasoning_effort (supports_reasoning_effort + reasoning_efforts allowlist) → extra_body["reasoning_effort"]. Supported by: k3 / k3-256k (low/high/max), both DeepSeek V4 models (low/medium/high/max), glm-5.2 (high/max), qwen3.8-max (low/medium/high/max). Invalid values are rejected client-side (mesh.py:604-609).

Interactions:

Auto-enable: requesting an effort on a toggle-capable model without an explicit thinking choice auto-enables thinking (adapter.py:136-154).
Kimi temperature coupling: kimi-for-coding presets lock temperature — 1.0 when thinking is on/omitted, 0.6 when thinking is explicitly disabled (temperature_thinking_disabled, config.py:259-263; adapter.py:76-90). Kimi presets also pin top_p=0.95.
Fail-open strip: if a provider answers 400 and the error text points at thinking/reasoning_effort (or image input), the adapter retries once with those params stripped (adapter.py:179-249).
K3 always reasons — it has no toggle; only the effort level is adjustable (config.py:329-332).

verified against code: 2026-08-09