Service-oriented Multi-model Management Framework (SMMF)

Relevant source files

• .mypy.ini
• README.ja.md
• README.md
• README.zh.md
• assets/LOGO_SMALL.png
• assets/Twitter_LOGO.png
• assets/Youtube_LOGO.png
• assets/ding.jpg
• assets/wechat.jpg
• docs/docs/modules/smmf.md
• docs/docs/upgrade/v0.5.1.md
• examples/rag/embedding_rag_example.py
• examples/rag/rag_embedding_api_example.py
• packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py

Purpose and Scope

The Service-oriented Multi-model Management Framework (SMMF) is DB-GPT's unified infrastructure for managing and deploying Large Language Models (LLMs). SMMF provides a consistent interface for interacting with 50+ models through both local deployment and API proxies, abstracting away the complexities of different inference frameworks, hardware requirements, and API protocols.

This document covers the architecture, components, deployment strategies, and configuration of SMMF. For information about how models are used in RAG pipelines, see RAG Pipeline and Knowledge Management. For model configuration in Docker deployments, see Docker Base Image and Build System. For acceleration-specific implementation details, see Hardware Acceleration and Performance.

Sources: README.md1-363 docs/docs/modules/smmf.md1-158

---

Overview

SMMF addresses the challenge that there is no de facto standard for deploying and serving LLMs. New models are constantly released with different requirements, training methods, and inference frameworks. Without SMMF, applications would need to implement custom adapters for each model, significantly increasing development complexity and maintenance burden.

SMMF solves this by providing:

• Unified Model Interface: Single API for all models regardless of deployment type
• Flexible Deployment: Support for local inference (HuggingFace Transformers, vLLM, LLAMA.cpp, MLX) and remote APIs (OpenAI, DeepSeek, Qwen, Ollama, etc.)
• Worker-based Architecture: Scalable model serving with load balancing and lifecycle management
• Hardware Optimization: Integrated acceleration through Flash Attention, quantization (BitsAndBytes, GPTQ), and multi-GPU support
• Configuration-driven: Model registration and deployment via .toml configuration files

Sources: README.md55-183 docs/docs/modules/smmf.md1-30

---

Architecture Components

High-Level Architecture

Sources: README.md180-292 Diagram 3 from high-level architecture

---

Component Descriptions

Model Registry

The Model Registry maintains metadata about available models, including their type (local/proxy), configuration parameters, and deployment status. It serves as the central source of truth for model discovery and selection.

Key Responsibilities:

• Register models from .toml configuration files
• Store model metadata (name, type, path/endpoint, parameters)
• Provide model lookup and filtering capabilities
• Track model availability and health status

Configuration Format: Models are registered via TOML files in two categories:

• dbgpt-local-*.toml: Local models deployed using inference frameworks
• dbgpt-proxy-*.toml: Remote API models accessed through proxy adapters

Sources: docs/docs/modules/smmf.md32-80

Model Controller

The Model Controller manages the complete lifecycle of model workers, from initialization to shutdown. It coordinates between the registry, worker manager, and individual workers.

Key Responsibilities:

• Initialize and start model workers based on configuration
• Monitor worker health and availability
• Handle worker failures and recovery
• Coordinate shutdown and cleanup

Sources: docs/docs/modules/smmf.md82-105

Worker Manager

The Worker Manager handles the operational aspects of the worker pool, including load distribution and resource allocation.

Key Responsibilities:

• Maintain pool of active workers
• Route inference requests to appropriate workers
• Implement load balancing strategies
• Monitor worker performance and capacity
• Scale workers up/down based on demand

Load Balancing: The Worker Manager distributes requests across available workers using strategies such as round-robin, least-connections, or weighted distribution based on worker capacity.

Sources: docs/docs/modules/smmf.md107-130

Model Workers

Model Workers are the execution units that host and serve models. Each worker encapsulates a model instance and its associated inference framework.

Worker Types:

Worker Type	Inference Framework	Use Case
HuggingFace Worker	Transformers library	General-purpose local inference
vLLM Worker	vLLM engine	High-throughput production serving
LLAMA.cpp Worker	LLAMA.cpp	CPU/Metal inference, reduced memory
MLX Worker	Apple MLX	Optimized for Apple Silicon
Proxy Worker	API client	Remote model access via APIs

Sources: README.md180-292 Diagram 3 from high-level architecture

---

Deployment Strategies

Local Model Deployment

Local deployment runs models on the same infrastructure as the DB-GPT application, providing full control over model execution, data privacy, and performance tuning.

HuggingFace Transformers

The default local deployment option using the HuggingFace Transformers library.

Characteristics:

• Supports widest range of model architectures
• Flexible configuration for precision (FP32, FP16, INT8, INT4)
• Compatible with quantization methods (BitsAndBytes, GPTQ)
• Moderate memory and compute requirements

Configuration Parameters:

• model_name: HuggingFace model identifier or local path
• model_type: Specific model architecture (e.g., llama, qwen, chatglm)
• device: Target device (cuda, cpu)
• load_in_8bit, load_in_4bit: Quantization flags

Sources: README.md180-183

vLLM Deployment

vLLM provides high-throughput, low-latency serving for production workloads through continuous batching and optimized memory management.

Characteristics:

• Significantly higher throughput than HuggingFace (2-3x)
• Efficient memory usage via PagedAttention
• Automatic batching of requests
• Native tensor parallelism for multi-GPU

Use Cases:

• Production API serving with high request volume
• Multi-user concurrent access
• Latency-sensitive applications

Configuration Parameters:

• max_model_len: Maximum sequence length
• gpu_memory_utilization: Fraction of GPU memory to use
• tensor_parallel_size: Number of GPUs for tensor parallelism
• quantization: Quantization method (awq, gptq)

Sources: README.md180-183 Diagram 3 from high-level architecture

LLAMA.cpp

CPU-optimized inference engine with optional Metal acceleration for Apple devices.

Characteristics:

• Runs on CPU with acceptable performance
• Low memory footprint
• Metal GPU support for M1/M2/M3 Macs
• Quantized model formats (GGUF)

Use Cases:

• CPU-only environments
• Development and testing on laptops
• Memory-constrained deployments
• Apple Silicon optimization

Configuration Parameters:

• model_path: Path to GGUF format model
• n_ctx: Context window size
• n_gpu_layers: Number of layers to offload to GPU (Metal)
• n_threads: CPU thread count

Sources: README.md180-183 Diagram 3 from high-level architecture

MLX

Apple's machine learning framework optimized for Apple Silicon (M1/M2/M3).

Characteristics:

• Native unified memory architecture support
• Optimized kernels for Apple Silicon
• Low latency on Mac devices
• Efficient memory usage

Use Cases:

• Development on Apple Silicon Macs
• Inference on Apple devices
• Unified memory models (shared CPU/GPU memory)

Sources: README.md180-183 Diagram 3 from high-level architecture

---

API Proxy Deployment

API proxy deployment connects to remote LLM services through standardized adapters, enabling access to commercial models without local infrastructure.

Supported Providers

Provider	Models Supported	Configuration Key
OpenAI	GPT-4, GPT-3.5, GPT-4-Turbo	openai_api_key, openai_api_base
DeepSeek	DeepSeek-V3, DeepSeek-R1, DeepSeek-Coder	deepseek_api_key
Qwen	Qwen-2.5, QwQ-32B	qwen_api_key
Ollama	Any Ollama-served model	ollama_api_base
Baidu (Wenxin)	ERNIE models	wenxin_api_key, wenxin_secret_key
Alibaba (Tongyi)	Qwen series	tongyi_api_key
Zhipu (ChatGLM)	GLM-4, GLM-Z1	zhipu_api_key

Sources: README.md184-318 README.zh.md195-323

Proxy Adapter Architecture

Each provider has a dedicated adapter implementing a common interface:

Adapter Responsibilities:

• Translate DB-GPT request format to provider-specific API
• Handle authentication and rate limiting
• Normalize responses to unified format
• Implement retry logic and error handling
• Support streaming responses where available

Configuration Parameters (common):

• proxy_server_url: Base URL for API
• proxy_api_key: Authentication key
• proxy_api_version: API version (if applicable)
• proxies: HTTP proxy configuration

Sources: docs/docs/modules/smmf.md32-80

---

Configuration System

Configuration File Structure

SMMF uses TOML format for model configuration, separated into local and proxy configurations.

Local Model Configuration

Example structure for dbgpt-local-llama.toml:

Key Sections:

• [model]: Model identification and location
• [deployment]: Hardware allocation
• [optimization]: Quantization and acceleration
• [inference]: Generation parameters

Sources: Inferred from docs/docs/modules/smmf.md1-158 and standard TOML configuration patterns

Proxy Model Configuration

Example structure for dbgpt-proxy-openai.toml:

Key Sections:

• [proxy]: Provider identification
• [authentication]: API credentials
• [parameters]: Provider-specific parameters

Sources: Inferred from docs/docs/modules/smmf.md1-158

Configuration Loading Process

Sources: docs/docs/modules/smmf.md82-105

---

Model Lifecycle Management

Worker Initialization

The initialization process loads models into memory and prepares workers for serving requests.

Initialization Steps:

1. Configuration Validation: Verify model path, credentials, and parameters
2. Resource Allocation: Reserve GPU memory, CPU cores, or API quota
3. Model Loading: Load weights into memory or establish API connection
4. Warmup: Execute test inference to ensure readiness
5. Registration: Register with Worker Manager as available

Sources: docs/docs/modules/smmf.md107-130

Request Routing

Routing Strategies:

• Round-Robin: Distribute requests evenly across workers
• Least-Connections: Route to worker with fewest active requests
• Weighted: Prioritize workers based on capacity (GPU memory, throughput)
• Model-Specific: Route to workers serving specific models

Sources: docs/docs/modules/smmf.md107-130

Health Monitoring

Workers report health metrics to the Controller:

Metric	Description	Threshold
Response Time	Average inference latency	Alert if > 5s
Error Rate	Failed requests / total requests	Alert if > 5%
Memory Usage	GPU/CPU memory consumption	Alert if > 90%
Queue Length	Pending requests	Alert if > 100
Heartbeat	Last successful ping	Mark unhealthy if > 30s

Unhealthy workers are removed from the pool and optionally restarted.

Sources: Inferred from docs/docs/modules/smmf.md82-130

---

Hardware Acceleration

SMMF integrates multiple acceleration techniques to optimize inference performance. These are implemented in the packages/dbgpt-accelerator package.

Flash Attention

Flash Attention is an optimized attention mechanism that reduces memory usage and improves speed through kernel fusion and memory hierarchy optimization.

Benefits:

• 2-4x faster attention computation
• Reduced memory footprint (enables longer sequences)
• Compatible with vLLM and HuggingFace

Supported Models: Most transformer-based architectures (Llama, Qwen, ChatGLM, etc.)

Configuration: Enabled by default in vLLM; requires flash-attn package for HuggingFace

Sources: README.md126-127 Diagram 3 from high-level architecture

Quantization

Quantization reduces model precision to lower memory usage and increase throughput.

BitsAndBytes

Precision Options:

• 8-bit: ~50% memory reduction, minimal quality loss
• 4-bit: ~75% memory reduction, acceptable quality loss

Configuration:

Use Cases: Memory-constrained environments, consumer GPUs

Sources: README.md126-127 Diagram 3 from high-level architecture

GPTQ

GPTQ (Generative Pre-trained Transformer Quantization) performs post-training quantization optimized for generation tasks.

Characteristics:

• 3-4 bit weights with minimal perplexity increase
• Requires pre-quantized model weights
• Better quality than standard quantization at same bit depth

Configuration:

Sources: README.md126-127 Diagram 3 from high-level architecture

CUDA Support

SMMF supports multiple CUDA versions to accommodate different GPU driver environments:

CUDA Version	Compatible GPUs	Status
11.8	RTX 30xx, A100, V100	Supported
12.1	RTX 40xx, H100	Supported
12.4	Latest GPUs	Supported

Installation: CUDA-specific wheels are available for torch, vllm, and acceleration libraries.

Sources: README.md126-127 Diagram 3 from high-level architecture

---

Inference Backends

vLLM Backend

vLLM is the recommended backend for production deployments requiring high throughput.

Architecture:

• PagedAttention: Dynamic memory allocation for KV cache
• Continuous Batching: Automatic request batching
• Tensor Parallelism: Multi-GPU model sharding
• Sequence Parallelism: Distribute sequences across GPUs

Performance: 2-3x higher throughput than HuggingFace Transformers

Sources: Diagram 3 from high-level architecture

Text Generation Inference (TGI)

HuggingFace's optimized serving framework.

Characteristics:

• Production-ready serving
• gRPC and REST APIs
• Token streaming
• Dynamic batching

Sources: Diagram 3 from high-level architecture

TensorRT

NVIDIA's high-performance inference engine.

Characteristics:

• Kernel optimization for NVIDIA GPUs
• Layer fusion and precision calibration
• Lowest latency for supported models
• Requires model conversion to TensorRT format

Sources: Diagram 3 from high-level architecture

---

Model Registry Implementation

The Model Registry is the central component that tracks all available models and their metadata.

Registry Data Structure

Key Operations:

• register_model(): Add new model from configuration
• get_model(): Retrieve model metadata by name
• list_models(): Query models with filters (type, status, provider)
• update_status(): Update model availability

Sources: docs/docs/modules/smmf.md32-80

---

Integration with Other Systems

Integration with RAG Pipeline

SMMF provides model inference for RAG operations:

Model Types Used:

• Embedding Models: Convert text to vectors (e.g., bge-large-en-v1.5)
• Generation Models: Produce responses from prompts and context

For details on RAG integration, see RAG Pipeline and Knowledge Management.

Sources: examples/rag/embedding_rag_example.py1-75

Integration with Multi-Agent Framework

Agents use SMMF for LLM inference during planning and execution:

• Planning: Generate action plans from current state
• Tool Selection: Decide which tools to invoke
• Response Generation: Produce final answers

For details on agent integration, see Multi-Agents and AWEL Workflows.

Sources: Diagram 4 from high-level architecture

Integration with GBI (Text2SQL)

SMMF provides the LLM for natural language to SQL translation:

1. Parse natural language query
2. Generate SQL using LLM via SMMF
3. Validate and execute SQL
4. Format results

For details on GBI, see Generative Business Intelligence (GBI).

Sources: Diagram 6 from high-level architecture

---

Database Storage

Model metadata and operational state are persisted using SQLAlchemy.

Storage Schema

Models are stored in the metadata database with the following structure:

Column	Type	Description
id	Integer	Primary key
model_name	String	Unique model identifier
model_type	String	Model architecture (llama, qwen, etc.)
deployment_type	Enum	Local or Proxy
config_json	JSON	Full configuration
status	Enum	Ready, Starting, Failed, etc.
created_at	DateTime	Registration timestamp
updated_at	DateTime	Last status update

Storage Interface: SQLAlchemyStorage provides CRUD operations

Sources: packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py1-137

---

Error Handling and Resilience

Worker Failure Recovery

Failure Scenarios:

• Out of Memory: Reduce batch size, reload with lower precision
• Model Load Failure: Verify model files, check disk space
• API Rate Limiting: Implement exponential backoff, rotate keys
• Network Errors: Retry with timeout, failover to backup endpoint

Sources: Inferred from docs/docs/modules/smmf.md82-130

Request Timeout Handling

SMMF implements timeouts at multiple levels:

Timeout Type	Default	Configurable
Connection	10s	Yes
Inference	60s	Yes
Queue Wait	120s	Yes

Requests exceeding timeouts return error responses to prevent resource exhaustion.

Sources: Inferred from system design principles

---

Performance Optimization

Batching Strategy

SMMF supports request batching to increase throughput:

Static Batching: Accumulate requests for fixed time window (e.g., 100ms) Dynamic Batching (vLLM): Continuously add/remove requests based on sequence length

Configuration:

Sources: Inferred from vLLM integration

Caching

Model Weight Caching: Avoid reloading frequently-used models KV Cache: Store key-value pairs for efficient autoregressive generation Response Caching: Cache responses for duplicate queries (optional)

Sources: Inferred from system design

Monitoring Metrics

Metric	Purpose	Export Method
Requests/sec	Throughput monitoring	Prometheus
P50/P95/P99 Latency	Performance tracking	Prometheus
GPU Utilization	Resource optimization	NVIDIA-SMI
Memory Usage	Capacity planning	Prometheus
Error Rate	Reliability monitoring	Prometheus

Sources: Inferred from production best practices

---

Summary

SMMF provides a comprehensive framework for managing heterogeneous LLM deployments through:

1. Unified Interface: Single API abstracting 50+ models across local and remote deployments
2. Flexible Architecture: Worker-based design supporting multiple inference frameworks
3. Configuration-Driven: TOML-based model registration simplifying deployment
4. Production-Ready: Load balancing, health monitoring, and failure recovery
5. Performance Optimized: Hardware acceleration, quantization, and efficient batching
6. Extensible: Plugin architecture for new models and providers

This architecture enables DB-GPT to support rapid model iteration and diverse deployment scenarios while maintaining consistent application behavior.

Sources: README.md1-363 docs/docs/modules/smmf.md1-158 Diagrams 1, 3 from high-level architecture