File Storage and S3 Integration

Relevant source files

• docker-compose/local/.env.example
• docker-compose/local/.env.zh-CN.example
• docker-compose/local/bucket.config.json
• docker-compose/local/docker-compose.yml
• docker-compose/local/init_data.json
• docker-compose/setup.sh
• docs/development/database-schema.dbml
• docs/self-hosting/server-database/docker-compose.mdx
• docs/self-hosting/server-database/docker-compose.zh-CN.mdx
• docs/self-hosting/server-database/docker.mdx
• docs/self-hosting/server-database/docker.zh-CN.mdx
• docs/self-hosting/server-database/zeabur.mdx
• docs/self-hosting/server-database/zeabur.zh-CN.mdx
• packages/const/src/userMemory.ts
• packages/database/migrations/0041_improve_index.sql
• packages/database/migrations/0063_add_columns_for_several_tables.sql
• packages/database/migrations/0067_add_agent_cron_tables.sql
• packages/database/migrations/0072_add_market_identifier_chat_group.sql
• packages/database/migrations/meta/0041_snapshot.json
• packages/database/migrations/meta/0063_snapshot.json
• packages/database/migrations/meta/0067_snapshot.json
• packages/database/migrations/meta/0072_snapshot.json
• packages/database/migrations/meta/_journal.json
• packages/database/src/models/agentCronJob.ts
• packages/database/src/models/document.ts
• packages/database/src/models/file.ts
• packages/database/src/schemas/_helpers.ts
• packages/database/src/schemas/agent.ts
• packages/database/src/schemas/chatGroup.ts
• packages/database/src/schemas/file.ts
• packages/database/src/schemas/index.ts
• packages/database/src/schemas/message.ts
• packages/database/src/schemas/rag.ts
• packages/database/src/schemas/relations.ts
• packages/database/src/schemas/session.ts
• packages/database/src/schemas/topic.ts
• packages/database/src/schemas/user.ts
• packages/database/src/utils/columns.ts
• packages/database/src/utils/idGenerator.ts
• packages/prompts/src/prompts/messagesToText/__snapshots__/index.test.ts.snap
• packages/prompts/src/prompts/messagesToText/index.test.ts
• packages/prompts/src/prompts/messagesToText/index.ts
• src/server/routers/lambda/userMemory.ts

This document covers LobeChat's file storage architecture and S3/MinIO integration, including upload flows, deduplication strategies, and deployment configurations. For information about the RAG pipeline that processes these files, see 5.2 Chunking and Embedding Pipeline. For database models related to files, see 6.2 Core Data Models.

Architecture Overview

LobeChat uses S3-compatible object storage for persisting uploaded files and documents. The system supports self-hosted RustFS deployments (default in Docker Compose) and cloud-based S3 services (AWS S3, Cloudflare R2, etc.).

Diagram: File Storage System Architecture

Sources: docker-compose/local/docker-compose.yml56-94 packages/database/src/models/file.ts19-40 docs/development/database-schema.dbml354-395

S3 Configuration

Environment Variables

LobeChat requires the following S3-related environment variables:

Variable	Required	Description	Example
S3_ENDPOINT	Yes	S3 API endpoint URL	http://localhost:9000
S3_PUBLIC_DOMAIN	Yes	Public URL for file access	http://localhost:9000
S3_BUCKET	Yes	Bucket name for file storage	lobe
S3_ACCESS_KEY_ID / S3_ACCESS_KEY	Yes	S3 access key	admin
S3_SECRET_ACCESS_KEY	Yes	S3 secret key	YOUR_RUSTFS_PASSWORD
S3_ENABLE_PATH_STYLE	No	Use path-style URLs (for RustFS)	1
S3_SET_ACL	No	Set object ACL on upload	0
LLM_VISION_IMAGE_USE_BASE64	No	Use base64 for vision models	1

RustFS-Specific Variables

When using the default RustFS deployment in Docker Compose:

Variable	Required	Description	Example
RUSTFS_ACCESS_KEY	Yes	RustFS access key	admin
RUSTFS_SECRET_KEY	Yes	RustFS secret key	YOUR_RUSTFS_PASSWORD
RUSTFS_PORT	No	RustFS API port	9000
RUSTFS_LOBE_BUCKET	Yes	Bucket name in RustFS	lobe

Sources: docker-compose/local/.env.example16-43 docker-compose/local/docker-compose.yml56-71

File Storage Strategy

Two-Tier Storage Model

LobeChat implements a two-tier file storage strategy to optimize storage and enable deduplication:

Global Files Table

The global_files table stores the actual file metadata and S3 references. Files are deduplicated by their SHA-256 hash.

Key fields:

• hash_id: SHA-256 hash of file content (primary key)
• url: S3 object URL
• size: File size in bytes
• file_type: MIME type
• metadata: Additional file metadata (JSONB)
• creator: User ID who first uploaded the file
• created_at: Creation timestamp
• accessed_at: Last access timestamp

Files Table

The files table contains user-specific references to global files, enabling:

• User-level access control
• Knowledge base associations
• Custom metadata per user
• Deletion without affecting other users

Key fields:

• id: Unique file reference ID (text)
• user_id: Owner of this file reference
• file_hash: Reference to global_files.hash_id
• name: User-defined file name
• url: S3 object URL (denormalized for performance)
• size: File size in bytes (denormalized)
• file_type: MIME type (denormalized)
• source: Source path or identifier
• parent_id: Optional parent file for hierarchy
• metadata: Custom metadata (JSONB)
• chunk_task_id: Reference to async chunking task
• embedding_task_id: Reference to async embedding task

Sources: docs/development/database-schema.dbml354-395 packages/database/src/schemas/file.ts26-44

Upload and Storage Flow

File Upload Process

Resource Item Structure

The frontend uses a unified ResourceItem interface for both files and documents:

Sources: src/types/resource.ts6-55 src/store/file/slices/resource/initialState.ts1-67

File Access and Public URLs

URL Configuration

Files are accessed through public URLs configured via S3_PUBLIC_DOMAIN. This allows:

• Direct browser access to files
• Image embedding in messages
• LLM provider access for vision models

Redis Caching Layer

LobeChat uses Redis to cache presigned URLs and improve file access performance:

Redis configuration for LobeChat:

• REDIS_URL=redis://redis:6379
• REDIS_PREFIX=lobechat
• REDIS_TLS=0 (disable TLS for internal network)

Sources: docker-compose/local/docker-compose.yml38-53 docker-compose/local/docker-compose.yml160-162

Path-Style vs Virtual-Hosted URLs

LobeChat supports both S3 URL styles:

Path-Style (recommended for RustFS):

http://localhost:9000/lobe/object-key

Enable with: S3_ENABLE_PATH_STYLE=1

Virtual-Hosted Style:

https://bucket-name.s3.amazonaws.com/object-key

Sources: docker-compose/local/docker-compose.yml153

RustFS Deployment in Docker Compose

RustFS Service Configuration

Diagram: RustFS Docker Compose Architecture

Sources: docker-compose/local/docker-compose.yml56-94

RustFS Container Configuration

The RustFS service is configured with the following settings:

Key features:

• Health check: RustFS exposes a /health endpoint for monitoring
• Console: Web UI enabled on port 9001
• Network mode: Shares network with network-service container
• Persistent storage: Data stored in rustfs-data volume

Sources: docker-compose/local/docker-compose.yml56-71

Bucket Initialization

The rustfs-init service uses MinIO Client (mc) to initialize the bucket on startup:

This initialization script:

1. Waits for RustFS to be healthy
2. Configures mc alias pointing to RustFS
3. Creates the lobe bucket if it doesn't exist
4. Applies public access policy from bucket.config.json

Sources: docker-compose/local/docker-compose.yml73-94 docker-compose/local/bucket.config.json1-23

Bucket Access Policy

The bucket.config.json file defines the public access policy for the bucket:

This policy allows:

• Public read access to all objects
• Bucket location and listing operations
• Put/Delete operations (should be restricted to authenticated users in production)

Sources: docker-compose/local/bucket.config.json1-23

Deduplication Strategy

Hash-Based Deduplication

LobeChat implements content-addressable storage using SHA-256 hashing:

Storage Savings

Deduplication provides:

• Reduced storage costs: Identical files stored once
• Faster uploads: Duplicate files skip S3 upload
• Improved performance: Hash comparison instead of byte comparison

Deletion Behavior

When a user deletes a file:

1. The files table reference is deleted
2. The global_files entry remains (other users may reference it)
3. A background job can clean orphaned global_files entries

Sources: README.md406-414

Integration with RAG Pipeline

File Processing Flow

After upload, files proceed through the RAG pipeline:

Async Task Status Tracking

Files have processing status fields tracked through the ResourceItem interface:

• chunkingStatus: Status of document chunking
• chunkTaskId: Async task ID for chunking
• chunkingError: Error details if chunking failed
• embeddingStatus: Status of embedding generation
• embeddingTaskId: Async task ID for embedding
• embeddingError: Error details if embedding failed

The frontend polls for status updates during processing:

Sources: src/features/ResourceManager/components/Explorer/useCheckTaskStatus.ts1-29 src/types/resource.ts20-30

Cloud S3 Providers

AWS S3 Configuration

For AWS S3, configure:

Cloudflare R2 Configuration

For Cloudflare R2:

Alternative S3 Providers

LobeChat's S3 integration is compatible with any S3-compatible storage provider:

Diagram: S3 Provider Options

Sources: docker-compose/local/.env.example35-43 docker-compose/local/docker-compose.yml56-71

Bucket Initialization and Policies

Bucket Creation

The Docker Compose setup automatically creates the bucket on first run using MinIO Client (mc) in the rustfs-init service:

The initialization process:

1. Configures mc to connect to RustFS
2. Creates the lobe bucket (skips if exists)
3. Applies the public access policy from bucket.config.json

Sources: docker-compose/local/docker-compose.yml82-90

Public Access Policy

The bucket policy is defined in bucket.config.json and allows:

• s3:GetBucketLocation - Read bucket location
• s3:ListBucket - List objects in bucket
• s3:GetObject - Download objects
• s3:PutObject - Upload objects
• s3:DeleteObject - Delete objects

For production deployments, restrict write operations to authenticated users only:

Sources: docker-compose/local/bucket.config.json1-23

Troubleshooting

Health Check Validation

The LobeChat container performs health checks on startup to verify S3 connectivity:

Sources: docker-compose/local/docker-compose.yml194-203

Common Issues

Issue	Cause	Solution
Files not accessible	Wrong S3_PUBLIC_DOMAIN	Ensure matches reverse proxy config and RustFS endpoint
Upload fails	Bucket doesn't exist	Check rustfs-init service logs, verify bucket creation
403 Forbidden	Incorrect access keys	Verify RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY match in all services
Connection refused	RustFS not started	Check RustFS health: docker logs lobe-rustfs
Redis connection failed	Redis not running	Verify Redis service: docker logs lobe-redis
Bucket policy not applied	rustfs-init failed	Check init logs: docker logs lobe-rustfs-init

Sources: docker-compose/local/docker-compose.yml194-203 docker-compose/local/docker-compose.yml73-94

Security Considerations

Access Control

• Private buckets: Files are stored in a private bucket with public read policy
• Signed URLs: Consider implementing time-limited signed URLs for sensitive files
• User isolation: The files table provides user-level access control

Credentials Management

• Store S3 credentials in .env file (not committed to git)
• Use IAM roles when deploying to AWS
• Rotate access keys periodically
• Use read-only keys for application access when possible

Sources: docker-compose/local/.env.example35-38