LLM Architecture Wiki
How the multi-provider LLM orchestration works with priority-based fallback
Overview
The system uses an **LLM Orchestrator** to route all AI requests (summarization, translation, dating enrichment, conflict resolution) through a unified interface that supports multiple LLM providers.
Instead of being locked into a single provider, the orchestrator tries providers in a configured priority order. If the primary provider fails (API error, timeout, rate limit, billing issue), it automatically falls back to the next available provider.
This architecture provides **high availability**, cost optimization, and makes it trivial to add new providers in the future without modifying any business logic.
Provider Priority
The following providers are currently configured, attempted in order of priority:
| Priority | Provider | Model | Role |
|---|---|---|---|
| #1 | Anthropic | claude-opus-4-7 | Primary provider |
| #2 | OpenAI | gpt-5.4 | Fallback provider |
Execution Flow
When a service (e.g., BiographySummarizationService) needs to call an LLM, the following steps occur:
- The service builds an LlmRequest with systemPrompt, userContent, temperature, and a label for logging.
- The service calls
llmOrchestrator.execute(request). - The orchestrator filters providers that have an API key configured (isAvailable = true).
- It attempts the highest-priority available provider first (Anthropic).
- If the call succeeds, it returns an
LlmResponsewith the content, provider name, and model used. - If it fails, the error is logged and the next provider is attempted. If all fail, an exception is thrown.
Architecture Diagram
Visual representation of the request flow through the orchestration layer:
Priority-based routing + fallback
Claude Opus 4.7
GPT-5.4 (Fallback)
Request Scenarios
Scenario A — Primary Provider Succeeds
Request received
Service sends LlmRequest to the orchestrator
Anthropic called
Rate limiter enforces pacing, request sent to Claude Opus 4.7
Success
Response returned directly to the service. OpenAI is never called.
Scenario B — Fallback to Secondary Provider
Request received
Service sends LlmRequest to the orchestrator
Anthropic fails
429 rate limit after 5 retries, or timeout, or API error
Fallback triggered
Error logged, orchestrator moves to OpenAI provider
OpenAI succeeds
Response returned to the service from fallback provider
Error Handling
If **all providers fail**, the orchestrator throws LlmAllProvidersFailedException containing the error from each provider. The calling service handles this gracefully.
Each provider has its own rate limiter with exponential backoff + jitter to avoid thundering herd problems. Non-retryable errors (billing, authentication) skip retries immediately.
Non-Retryable Errors
Billing issues (billing_not_active), invalid API keys, and account deactivation are detected immediately and skip to the next provider without wasting retry attempts.
Graceful Degradation by Service
- Biography summarization: returns original text without summarizing
- Translation: skips translation, returns null
- Dating enrichment: skips the record, continues with next
- Academic text extraction: falls back to heuristic regex extraction
Future Extensibility
The architecture follows the **Open/Closed Principle** — adding a new provider requires zero changes to existing services or the orchestrator logic.
- Create a new class implementing
LlmProviderinterface - Add the new enum value to
LlmProviderType - Register the provider in
Application.ktand add it to the orchestrator's provider list - Configure the API key via environment variable — done
Database-Driven Priority (Planned)
Provider priority is currently hardcoded in LlmConfig. A future enhancement will store priority configuration in a database table, allowing runtime changes without redeployment.