Core Framework and AWEL

Relevant source files

• README.ja.md
• README.md
• README.zh.md
• assets/LOGO_SMALL.png
• assets/Twitter_LOGO.png
• assets/Youtube_LOGO.png
• assets/ding.jpg
• assets/wechat.jpg
• docs/docs/upgrade/v0.5.1.md
• packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py

Purpose and Scope

This document describes the core framework abstractions and the AWEL (Agentic Workflow Expression Language) workflow engine that form the foundational layer of DB-GPT. The core framework, located in packages/dbgpt-core, provides reusable interfaces for storage, resource management, and serialization that are used throughout the system. AWEL provides a declarative, DAG-based workflow execution engine for orchestrating complex agentic workflows.

For information about the overall system architecture and package organization, see Package Structure and Dependencies. For details on multi-agent systems that leverage AWEL, see Multi-Agents and AWEL Workflows.

Core Framework Architecture

The core framework in packages/dbgpt-core provides foundational abstractions that enable modularity and extensibility throughout DB-GPT. This package defines interfaces and base implementations that other packages depend on, establishing contracts for storage, serialization, and resource management.

Core Framework Components and Dependencies

Sources: README.md1-363 packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py1-140

Storage Abstractions

The core framework defines generic storage interfaces that enable different persistence backends. The primary abstractions are:

Interface/Class	Purpose	Location
StorageInterface[T, D]	Generic storage contract with CRUD operations	dbgpt.core.interface.storage
ResourceIdentifier	Unique identifier for stored resources	dbgpt.core.interface.storage
QuerySpec	Query specification with conditions, limits, offsets	dbgpt.core.interface.storage
StorageItemAdapter[T, D]	Converts between domain objects (T) and storage format (D)	dbgpt.core.interface.storage
Serializer	Serialization/deserialization of objects	dbgpt.core

The StorageInterface provides methods for:

• save(data: T) - Persist a new item
• update(data: T) - Modify an existing item
• save_or_update(data: T) - Upsert operation
• load(resource_id: ResourceIdentifier, cls: Type[T]) - Retrieve by ID
• delete(resource_id: ResourceIdentifier) - Remove an item
• query(spec: QuerySpec, cls: Type[T]) - Query with conditions
• count(spec: QuerySpec, cls: Type[T]) - Count matching items

Sources: packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py10-16

SQLAlchemy Implementation

The framework provides SQLAlchemyStorage, a concrete implementation of StorageInterface backed by relational databases via SQLAlchemy. This enables metadata storage in SQLite, MySQL, PostgreSQL, and other SQL databases.

SQLAlchemy Storage Implementation Flow

The SQLAlchemyStorage class manages database interactions:

• Accepts either a database URL or an existing DatabaseManager instance
• Uses a StorageItemAdapter to convert between domain objects and SQLAlchemy models
• Provides session management through context managers
• Supports configurable query classes for extended functionality

Key implementation details from packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py31-140:

The adapter pattern (StorageItemAdapter) enables clean separation between domain models and persistence models. The adapter must implement:

• to_storage_format(item: T) -> D - Convert domain object to ORM model
• from_storage_format(item: D) -> T - Convert ORM model to domain object
• get_query_for_identifier(model_class, resource_id, session) - Build query for ID lookup

Sources: packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py31-140

AWEL (Agentic Workflow Expression Language)

AWEL is a declarative workflow engine that enables developers to define complex, multi-step processes as Directed Acyclic Graphs (DAGs). It provides the orchestration layer for agents, data processing pipelines, and service integration throughout DB-GPT.

AWEL Architecture

AWEL System Architecture and Integration

AWEL workflows are composed of:

1. Triggers - Entry points that initiate workflow execution (HTTP requests, scheduled tasks, events)
2. Operators - Processing nodes that transform, filter, join, or route data
3. DAG Definition - Declarative specification of operator connections and data flow
4. Executor - Runtime engine that manages task scheduling, parallelization, and state

Sources: README.md55-57 README.zh.md55-58

DAG-Based Workflow Execution

AWEL workflows are defined as DAGs where:

• Nodes represent operators (processing units)
• Edges represent data flow between operators
• Execution follows topological order with support for parallelization

The DAG structure ensures:

• No circular dependencies
• Clear data lineage
• Deterministic execution order
• Ability to parallelize independent branches

Example AWEL Workflow DAG

Key characteristics:

• Operators can run in parallel when they have no dependencies
• Data flows through typed channels between operators
• The executor handles error propagation and retry logic
• State can be persisted for long-running workflows

Sources: README.md55-57

Workflow Operators

AWEL provides a library of reusable operators for common workflow patterns:

Operator Type	Purpose	Example Use Case
MapOperator	Transform data 1:1	Parse JSON, call LLM, format output
JoinOperator	Combine multiple inputs	Merge retrieval results from multiple sources
BranchOperator	Conditional routing	Route to different agents based on query type
ReduceOperator	Aggregate multiple items	Combine scores from multiple retrievers
FilterOperator	Select items by predicate	Filter documents by relevance threshold

Operators support:

• Async execution for I/O-bound operations
• Streaming data for large datasets
• Error handling with configurable retry strategies
• Resource management (connection pooling, rate limiting)

Sources: README.md55-57

Triggers and Entry Points

Triggers define how workflows are initiated:

HTTP Trigger

• Exposes workflow as REST API endpoint
• Parses request body into workflow input
• Returns workflow output as HTTP response
• Example: /api/v1/awel/trigger/<workflow_id>

Schedule Trigger

• Executes workflow on cron-like schedule
• Useful for batch processing, maintenance tasks
• Example: Daily knowledge base synchronization

Event Trigger

• Listens for system events (message queue, webhook)
• Enables reactive workflows
• Example: Process new documents added to storage

Sources: README.md153-154

Integration with Multi-Agent Framework

AWEL serves as the orchestration backbone for multi-agent systems. Agents can be invoked as workflow operators, enabling:

• Agent Composition - Chain multiple specialized agents in sequence
• Parallel Agent Execution - Run multiple agents concurrently and merge results
• Conditional Agent Selection - Route to different agents based on task requirements
• Shared State Management - Agents share context through workflow state

Example workflow: A user query triggers a workflow that:

1. Uses a planning agent to decompose the query
2. Executes multiple data agents in parallel to fetch information
3. Uses a synthesis agent to combine results
4. Formats the final response

This integration is covered in detail in Multi-Agents and AWEL Workflows.

Sources: README.md76-77

Foundational Interfaces

Beyond storage, the core framework provides additional abstractions used throughout DB-GPT.

Resource Management

The ResourceIdentifier abstraction provides a universal way to reference stored items:

This enables:

• Type-safe resource references across storage backends
• Consistent ID generation strategies
• Support for composite keys and hierarchical identifiers

Query Specifications

The QuerySpec class encapsulates query parameters:

This abstraction allows storage implementations to optimize queries based on backend capabilities (e.g., database indexes, vector similarity search).

Sources: packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py112-132

Serialization

The Serializer interface enables pluggable serialization strategies:

• JSON serialization for API responses
• Pickle for Python object persistence
• Protocol Buffers for efficient binary encoding
• Custom formats for specific data types

Serializers are used by storage implementations to persist complex objects that don't map directly to database columns.

Sources: packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py9

Core Framework Usage in DB-GPT

The core framework abstractions are used extensively throughout DB-GPT:

Core Framework Usage Across Packages

1. RAG Service (RAG and Knowledge Services) uses:

   • SQLAlchemyStorage for document and chunk metadata
   • Storage interfaces for vector store abstractions
   • AWEL workflows for document ingestion pipelines

2. Evaluation Service (Evaluation and Benchmark Services) uses:

   • SQLAlchemyStorage for benchmark results and dataset metadata
   • Query specifications for filtering evaluation runs

3. Knowledge Service uses:

   • Storage interfaces to manage knowledge spaces
   • AWEL workflows for document processing and indexing

4. Vector Store Extensions (packages/dbgpt-ext) implement:

   • Storage interfaces for unified vector DB access
   • Adapter patterns for converting domain objects to vector embeddings

Sources: README.md68-81 packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py1-140

Package Structure

The core framework is organized in packages/dbgpt-core with the following key modules:

packages/dbgpt-core/
├── src/dbgpt/
│   ├── core/
│   │   ├── interface/
│   │   │   └── storage.py         # Storage interfaces
│   │   └── serializer.py          # Serialization utilities
│   ├── storage/
│   │   └── metadata/
│   │       ├── db_storage.py      # SQLAlchemy implementation
│   │       └── db_manager.py      # Database manager
│   └── awel/                      # AWEL workflow engine (implied)
└── pyproject.toml                 # Package metadata

Dependencies are managed through pyproject.toml, allowing selective installation. The core package has minimal dependencies, while optional extras enable specific features (database drivers, serialization formats).

For more details on the complete package structure and dependency management, see Package Structure and Dependencies.

Sources: README.md122-127 packages/dbgpt-core/src/dbgpt/storage/metadata/db_storage.py1-4