LLM Service and Provider Adapters

Relevant source files

• README.md
• README_EN.md
• docker-compose.yml
• docs/user/mcp-server.md
• docs/user/mcp-server_en.md
• env.local.example
• mkdocs/docs/zh/user/mcp-server.md
• packages/core/src/index.ts
• packages/core/src/services/image-model/defaults.ts
• packages/core/src/services/llm/adapters/modelscope-adapter.ts
• packages/core/src/services/llm/service.ts
• packages/core/src/services/model/defaults.ts
• packages/core/src/services/model/manager.ts
• packages/core/src/services/model/types.ts
• packages/core/src/utils/environment.ts
• packages/core/tests/unit/image/modelscope-adapter.test.ts
• packages/core/tests/unit/llm/modelscope-adapter.test.ts
• packages/core/tests/unit/model/defaults.test.ts
• packages/core/tests/unit/model/manager.test.ts
• packages/mcp-server/src/config/models.ts
• vercel.json

Purpose and Scope

This page documents the LLM Service and Provider Adapter architecture in the @prompt-optimizer/core package. It covers the adapter pattern implementation that abstracts interactions with different LLM providers (OpenAI, Gemini, Anthropic, DeepSeek, etc.), the registry system that manages adapters, and the LLMService class that orchestrates all text generation requests.

For information about model configuration management, see Model Configuration Management. For prompt optimization workflows that use the LLM service, see Prompt Optimization Service.

Sources: packages/core/src/services/llm/service.ts1-418 packages/core/src/services/llm/types.ts1-200 packages/core/src/services/llm/adapters/registry.ts1-250

---

Architecture Overview

The LLM service follows a three-tier architecture that separates business logic from provider-specific implementations using the adapter pattern.

System Component Diagram

Sources: packages/core/src/services/llm/service.ts19-30 packages/core/src/services/llm/adapters/registry.ts15-45

---

The Adapter Pattern

Purpose and Benefits

The adapter pattern abstracts provider-specific API implementations behind a uniform interface (ITextProviderAdapter), enabling the system to support any OpenAI-compatible or custom API without modifying core business logic.

Key Benefits:

• Provider Independence: Core services remain agnostic to API specifics
• Easy Extension: New providers require only implementing the adapter interface
• Consistent Error Handling: Unified error types across all providers
• Testability: Mock adapters for unit testing without real API calls

ITextProviderAdapter Interface

Sources: packages/core/src/services/llm/types.ts85-167 packages/core/src/services/model/types.ts14-33

Core Adapter Methods

Method	Purpose	Return Type
getProvider()	Returns provider metadata (ID, name, baseURL, CORS restrictions, etc.)	TextProvider
getModels()	Returns static model list known at compile time	TextModel[]
buildDefaultModel()	Creates fallback model metadata when dynamic fetch fails	TextModel
sendMessage()	Non-streaming text generation	Promise<LLMResponse>
sendMessageStream()	Streaming text generation with callbacks	Promise<void>
sendMessageStreamWithTools()	Streaming with tool/function calling support	Promise<void>
fetchModels()	Dynamically fetches available models from provider API	Promise<TextModel[]>

Sources: packages/core/src/services/llm/types.ts85-167

---

Provider Registry System

TextAdapterRegistry Implementation

The TextAdapterRegistry class manages adapter instances using the singleton pattern per provider. It provides lazy initialization, caching, and graceful fallback mechanisms.

Registry Architecture Diagram

Sources: packages/core/src/services/llm/adapters/registry.ts15-150

Registry API

Key Behaviors:

• Lazy Initialization: Adapters instantiated on first access, not at construction
• Singleton per Provider: Each provider has exactly one adapter instance
• Model Fetching with Fallback: Dynamic model fetching falls back to static models on failure
• Error Handling: Throws ModelConfigError for unknown providers

Sources: packages/core/src/services/llm/types.ts169-176 packages/core/src/services/llm/adapters/registry.ts22-95

Built-in Provider Adapters

Provider ID	Adapter Class	API Base URL	Dynamic Models	CORS Restricted
openai	OpenAIAdapter	https://api.openai.com/v1	Yes	No
gemini	GeminiAdapter	https://generativelanguage.googleapis.com/v1beta	Yes	No
anthropic	AnthropicAdapter	https://api.anthropic.com/v1	No	No
deepseek	DeepSeekAdapter	https://api.deepseek.com/v1	No	No
siliconflow	SiliconFlowAdapter	https://api.siliconflow.cn/v1	Yes	Yes
zhipu	ZhipuAdapter	https://open.bigmodel.cn/api/paas/v4	No	Yes
dashscope	DashScopeAdapter	https://dashscope.aliyuncs.com/compatible-mode/v1	No	Yes
openrouter	OpenRouterAdapter	https://openrouter.ai/api/v1	Yes	No
modelscope	ModelScopeAdapter	https://api-inference.modelscope.cn/v1	Yes	Yes
custom	Uses OpenAIAdapter	User-configured	No	Varies

Sources: packages/core/src/services/llm/adapters/registry.ts47-90 packages/core/src/services/model/defaults.ts10-29

---

LLMService Implementation

Core Service Responsibilities

The LLMService class orchestrates all LLM interactions by:

1. Validating incoming requests (messages, model config)
2. Retrieving appropriate adapters from the registry
3. Preparing runtime configuration with parameter merging
4. Delegating actual API calls to adapters
5. Handling errors uniformly

Request Flow Diagram

Sources: packages/core/src/services/llm/service.ts122-153 packages/core/src/services/llm/service.ts281-299

Key LLMService Methods

Non-Streaming Message Sending

Returns structured response with content, reasoningContent (for o1-style models), and metadata.

Streaming Message Sending

Streams tokens via callbacks: onToken, onComplete, onError. Supports reasoning content streaming for models with chain-of-thought capabilities.

Tool-Enabled Streaming

Enables function calling for models that support tools (OpenAI, Gemini).

Connection Testing

Validates API configuration by sending a simple test message. Allows disabled models to be tested (option { allowDisabled: true }).

Model List Fetching

Retrieves available models for a provider, merging static and dynamic models. Returns options in format { value: modelId, label: modelName } for UI dropdowns.

Sources: packages/core/src/services/llm/service.ts80-279

---

Runtime Configuration and Parameter Merging

Parameter Override Strategy

The system supports three levels of parameter configuration:

1. Default Parameter Values: Defined in TextModel.defaultParameterValues
2. User Custom Overrides: Stored in TextModelConfig.paramOverrides
3. Request-Time Overrides: Passed to specific API calls (future extension point)

Configuration Merge Flow

Sources: packages/core/src/services/llm/service.ts281-299 packages/core/src/services/model/parameter-utils.ts1-200

prepareRuntimeConfig Implementation

Key Points:

• customParamOverrides is deprecated but still read for backward compatibility
• paramOverrides is the unified field that contains all parameter overrides
• mergeOverrides() utility handles schema validation and type checking
• The merged config is passed to adapters, not stored back to database

Sources: packages/core/src/services/llm/service.ts281-299 packages/core/src/services/model/parameter-utils.ts50-120

---

Provider Adapter Implementations

OpenAI Adapter

The OpenAIAdapter serves as the base implementation for most providers due to widespread OpenAI API compatibility.

Key Features:

• Uses official openai npm package SDK
• Supports streaming via SSE (Server-Sent Events)
• Handles function/tool calling
• Supports reasoning content for o1-series models
• Dynamic model fetching via /v1/models endpoint

Static Models (as of implementation):

• gpt-4o, gpt-4o-mini (128K context, tools, reasoning)
• o1-preview, o1-mini (reasoning models, 128K context)

Sources: packages/core/src/services/llm/adapters/openai-adapter.ts1-300

Gemini Adapter

Uses Google's @google/generative-ai SDK.

Key Features:

• Native Gemini SDK integration (not OpenAI-compatible wrapper)
• Supports both text and vision models
• Function calling via tools parameter
• Multi-turn conversations with history
• Safety settings configuration

Static Models:

• gemini-2.0-flash-exp (1M context, tools)
• gemini-1.5-pro, gemini-1.5-flash (2M/1M context, multimodal)

Sources: packages/core/src/services/llm/adapters/gemini-adapter.ts1-400

Anthropic Adapter

Uses @anthropic-ai/sdk for Claude models.

Key Features:

• Native Anthropic SDK integration
• Supports streaming with stream: true
• System messages handled separately in system parameter
• Tool use support
• No dynamic model fetching (relies on static list)

Static Models:

• claude-3-7-sonnet (200K context, tools)
• claude-3-5-haiku, claude-3-opus (200K context)

Sources: packages/core/src/services/llm/adapters/anthropic-adapter.ts1-350

ModelScope Adapter

Extends OpenAIAdapter for ModelScope API (阿里云魔搭社区).

Key Features:

• OpenAI-compatible API at https://api-inference.modelscope.cn/v1
• Daily free quota: 2000 API calls
• CORS restricted (requires desktop app or proxy)
• Supports dynamic model fetching

Static Models:

• Qwen/Qwen3-Coder-480B-A35B-Instruct (131K context)

Sources: packages/core/src/services/llm/adapters/modelscope-adapter.ts1-85 README.md33-36

Custom Provider Support

The system supports unlimited custom providers via two mechanisms:

1. Single Custom Provider: Environment variables VITE_CUSTOM_API_KEY, VITE_CUSTOM_API_BASE_URL, VITE_CUSTOM_API_MODEL
2. Multi-Custom Providers: Pattern VITE_CUSTOM_API_*_<suffix> (e.g., VITE_CUSTOM_API_KEY_ollama, VITE_CUSTOM_API_BASE_URL_ollama, VITE_CUSTOM_API_MODEL_ollama)

Custom providers reuse OpenAIAdapter since most local/custom APIs follow OpenAI compatibility.

Sources: packages/core/src/services/model/defaults.ts70-94 packages/core/src/utils/environment.ts256-394 env.local.example42-66

---

Error Handling and Validation

Validation Layers

Sources: packages/core/src/services/llm/service.ts33-76

Error Types

Error Class	Code Prefix	Usage
RequestConfigError	LLM-*	Invalid configuration: missing API key, disabled model, malformed messages
APIError	LLM-*	External API failures: network errors, rate limits, invalid responses
ModelConfigError	MODEL-*	Model configuration issues: unknown provider, validation failures

Error Code Examples:

• LLM-001: Model provider cannot be empty
• LLM-002: Messages array cannot be empty
• LLM-003: Model is not enabled
• LLM-004: Connection test failed

Sources: packages/core/src/services/llm/errors.ts1-100 packages/core/src/constants/error-codes.ts20-40

---

Usage Patterns and Examples

Basic Text Generation

Streaming with Callbacks

Tool/Function Calling

Dynamic Model Fetching

Sources: packages/core/src/services/llm/service.ts80-279 packages/core/tests/unit/llm/service.test.ts1-400

---

Electron Environment Handling

In Electron desktop applications, the LLMService is replaced by ElectronLLMProxy to bypass renderer process restrictions and CORS limitations.

Proxy Architecture

Key Points:

• Renderer process cannot directly instantiate LLMService due to lack of Node.js APIs
• createLLMService() factory detects Electron environment and returns proxy
• All method calls are serialized and forwarded to main process via IPC
• Main process has unrestricted access to Node.js modules and network APIs

Sources: packages/core/src/services/llm/service.ts386-398 packages/core/src/services/llm/electron-proxy.ts1-150 packages/core/src/utils/environment.ts132-167

---

Testing and Mocking

Unit Test Strategy

VCR Testing for Real API Calls

The codebase uses a VCR (Video Cassette Recorder) pattern to record and replay real API interactions for integration tests without consuming API credits.

Sources: packages/core/tests/unit/llm/service.test.ts1-500 packages/core/tests/unit/llm/modelscope-adapter.test.ts1-202

---

Summary Table: Key Classes and Their Roles

Class/Interface	File Path	Primary Responsibility
ILLMService	packages/core/src/services/llm/types.ts42-84	Service interface defining all LLM operations
LLMService	packages/core/src/services/llm/service.ts22-379	Main service implementation orchestrating requests
ITextProviderAdapter	packages/core/src/services/llm/types.ts85-167	Adapter interface for provider-specific implementations
TextAdapterRegistry	packages/core/src/services/llm/adapters/registry.ts15-150	Singleton registry managing adapter instances
OpenAIAdapter	packages/core/src/services/llm/adapters/openai-adapter.ts1-300	OpenAI and compatible APIs adapter
GeminiAdapter	packages/core/src/services/llm/adapters/gemini-adapter.ts1-400	Google Gemini native SDK adapter
AnthropicAdapter	packages/core/src/services/llm/adapters/anthropic-adapter.ts1-350	Anthropic Claude native SDK adapter
ModelScopeAdapter	packages/core/src/services/llm/adapters/modelscope-adapter.ts1-85	ModelScope API adapter (OpenAI-compatible)
ElectronLLMProxy	packages/core/src/services/llm/electron-proxy.ts1-150	IPC proxy for Electron renderer process

Sources: packages/core/src/index.ts38-57