Library Synchronization

Relevant source files

• docker/hwaccel.ml.yml
• docker/hwaccel.transcoding.yml
• docs/docs/FAQ.mdx
• docs/docs/administration/backup-and-restore.md
• docs/docs/features/facial-recognition.md
• docs/docs/features/hardware-transcoding.md
• docs/docs/features/libraries.md
• docs/docs/features/ml-hardware-acceleration.md
• docs/docs/guides/assets/20231219_183007_add-new-server-option.png
• docs/docs/guides/better-facial-clusters.md
• docs/docs/guides/database-gui.md
• docs/docs/guides/remote-access.md
• docs/docs/guides/remote-machine-learning.md
• docs/docs/guides/template-backup-script.md
• docs/docs/overview/quick-start.mdx
• server/src/repositories/event.repository.ts
• server/src/services/job.service.spec.ts
• server/src/services/job.service.ts
• server/src/services/library.service.spec.ts
• server/src/services/library.service.ts
• server/src/services/map.service.spec.ts
• server/src/services/notification.service.spec.ts
• server/src/services/notification.service.ts
• server/src/services/smart-info.service.spec.ts
• server/src/services/smart-info.service.ts
• server/src/services/storage-template.service.spec.ts
• server/src/services/storage-template.service.ts
• server/src/services/system-config.service.ts

This page covers Immich's external library feature: how it scans import paths on the filesystem, applies exclusion patterns, detects new and changed files, tracks asset offline/online status, watches for filesystem events in real time, and handles library deletion.

This is distinct from the mobile client sync protocol (see Data Synchronization) and from the general asset processing pipeline that runs after a file is imported (see Asset Processing Pipeline).

---

Overview

An external library allows an Immich user to surface photos and videos that already exist on the filesystem — outside of Immich's managed upload area — without copying or moving them. The server tracks each file's path, monitors for changes, and keeps the asset database up to date as files appear, change, or disappear on disk.

All library synchronization logic is implemented in LibraryService (server/src/services/library.service.ts).

---

Library Data Model

A library record contains the following key fields that drive synchronization:

Field	Type	Description
importPaths	string[]	Absolute directory paths to crawl recursively
exclusionPatterns	string[]	Glob patterns matched against full file paths to skip
ownerId	string	The user who owns all assets in this library
refreshedAt	Date	Timestamp of the last completed file scan
deletedAt	Date | null	Set when soft-deleted; triggers cleanup

All import paths must be absolute, readable directories. They may not overlap with Immich's own managed storage path (enforced by StorageCore.isImmichPath()).

Default exclusion patterns applied to new libraries (server/src/services/library.service.ts226-233):

**/@eaDir/**
**/._*
**/#recycle/**
**/#snapshot/**
**/.stversions/**
**/.stfolder/**

---

Scan Architecture

A full library scan is split into two independent sub-scans that run in parallel:

1. File sync — crawls import paths on disk to find new files and import them as assets.
2. Asset sync — iterates existing assets in the database to detect offline, online, or modified files.

Diagram: Full Library Scan Job Chain

Sources: server/src/services/library.service.ts449-475 server/src/services/library.service.ts612-680 server/src/services/library.service.ts696-772

---

File Sync: Importing New Files

Handler: handleQueueSyncFiles (JobName.LibrarySyncFilesQueueAll)

1. Validates each importPath with validateImportPath() — checks existence, that it is a directory, and that it has read permission.
2. Calls storageRepository.walk() with the valid paths and the library's exclusionPatterns. This yields batches of file paths (JOBS_LIBRARY_PAGINATION_SIZE at a time).
3. For each batch, calls assetRepository.filterNewExternalAssetPaths(libraryId, batch) to exclude paths already in the database.
4. Queues a LibrarySyncFiles job for the new paths.
5. Updates library.refreshedAt when the crawl completes.

Handler: handleSyncFiles (JobName.LibrarySyncFiles)

For each path in the job, processEntity() is called:

• stat() is called to get mtime
• An AssetTable row is constructed with isExternal: true, deviceId: 'Library Import', and a SHA-1 checksum of path:<normalizedPath>
• Rows are inserted in chunks of 5000 via assetRepository.createAll()

After insertion, queuePostSyncJobs() queues a SidecarCheck job for each new asset ID, which chains into the standard asset processing pipeline (metadata extraction → thumbnails → ML jobs).

Diagram: handleSyncFiles to Asset Processing

Sources: server/src/services/library.service.ts238-277 server/src/services/library.service.ts395-428 server/src/services/job.service.ts70-72

---

Asset Sync: Detecting Offline and Modified Files

Handler: handleQueueSyncAssets (JobName.LibrarySyncAssetsQueueAll)

This handler performs a two-phase check:

Phase 1 — SQL fast-path: Calls assetRepository.detectOfflineExternalAssets(libraryId, importPaths, exclusionPatterns). This uses a database query to directly mark assets offline if their path falls outside all import paths or matches an exclusion pattern. This is efficient and avoids per-file stat() calls for clearly out-of-scope assets.

Phase 2 — Disk check: If not all assets were caught by Phase 1, streams all remaining asset IDs from libraryRepository.streamAssetIds() and queues them in batches as LibrarySyncAssets jobs.

Handler: handleSyncAssets (JobName.LibrarySyncAssets)

For each asset in the batch, calls storageRepository.stat(asset.originalPath) and passes the result to checkExistingAsset():

Diagram: Asset Sync Decision Logic (checkExistingAsset)

Sources: server/src/services/library.service.ts477-611 server/src/repositories/library.repository.ts

The AssetSyncResult enum values and transitions are defined in server/src/repositories/library.repository.ts

Offline vs. Trashed

When marking an asset offline, the service distinguishes between normal and trashed assets:

• Normal online asset goes offline: isOffline=true, deletedAt=new Date() — appears in trash in the UI.
• Trashed asset goes offline: isOffline=true, deletedAt unchanged — stays trashed.
• Offline asset comes back online (not trashed): isOffline=false, deletedAt=null — restored to timeline.
• Offline trashed asset comes back online: isOffline=false, deletedAt unchanged — stays in trash.

Sources: server/src/services/library.service.ts541-557

---

Periodic Scan (Cron Job)

On startup, onConfigInit (server/src/services/library.service.ts35-58) acquires DatabaseLock.Library to ensure only one microservice instance manages scheduling. If the lock is acquired:

• A cron job named CronJob.LibraryScan is registered via cronRepository.create() using the expression from library.scan.cronExpression (default: once daily).
• When the cron fires, it queues JobName.LibraryScanQueueAll, which triggers a full scan of all libraries.

When system configuration changes (onConfigUpdate), the cron expression and enabled state are updated via cronRepository.update().

The default scan schedule is 0 0 * * * (midnight daily), configurable in Administration → Settings → External Library.

Sources: server/src/services/library.service.ts35-77

---

Filesystem Watching (Experimental)

When library.watch.enabled is true and the current microservice holds DatabaseLock.Library, watchAll() is called, which calls watch(id) for each library.

The watch() method (server/src/services/library.service.ts79-152) sets up a filesystem watcher via storageRepository.watch() (backed by chokidar) with these settings:

Option	Value
usePolling	false (uses native inotify/FSEvents)
ignoreInitial	true (existing files not emitted on start)
awaitWriteFinish.stabilityThreshold	5000 ms
awaitWriteFinish.pollInterval	1000 ms

A picomatch matcher is built from all supported media extensions and the library's exclusionPatterns.

Events and resulting jobs:

Filesystem Event	Condition	Job Queued
add	path matches media type and not excluded	LibrarySyncFiles
change	path matches media type and not excluded	LibrarySyncFiles
unlink	any path	LibraryRemoveAsset

LibraryRemoveAsset (server/src/services/library.service.ts682-694) looks up the asset by library ID and original path, then removes it immediately via assetRepository.remove().

Note: File watching does not work reliably on network filesystems (NFS, CIFS/SMB). Periodic scanning must be used instead. An ENOSPC error indicates the system's inotify watch limit has been reached; increase fs.inotify.max_user_watches.

Sources: server/src/services/library.service.ts79-175

---

Library Deletion

Diagram: Library Deletion Flow

The handleDeleteLibrary job queues AssetDelete jobs with deleteOnDisk: false — the original files on disk are not deleted. Assets are removed from the database only.

If the deletion process is interrupted (e.g., server restart), the LibraryDeleteCheck job (run on each LibraryScanQueueAll) finds libraries with a non-null deletedAt via libraryRepository.getAllDeleted() and re-queues their LibraryDelete jobs to complete cleanup.

Sources: server/src/services/library.service.ts344-393 server/src/services/library.service.ts205-219

---

Configuration Reference

All library settings live under the library key in the system configuration:

Config Path	Default	Description
library.scan.enabled	true	Whether the periodic scan cron job is active
library.scan.cronExpression	'0 0 * * *'	Cron schedule for automatic scans
library.watch.enabled	false	Enable real-time filesystem watching

These are configurable via Administration → Settings → External Library in the web UI.

Sources: server/src/services/library.service.ts35-77

---

Job Summary

All library jobs run on QueueName.Library.

Job Name	Handler	Description
LibraryScanQueueAll	handleQueueScanAll	Triggers a full scan of all libraries; also queues LibraryDeleteCheck
LibrarySyncFilesQueueAll	handleQueueSyncFiles	Crawls import paths, finds new files, queues import jobs
LibrarySyncFiles	handleSyncFiles	Imports a batch of new file paths as asset records
LibrarySyncAssetsQueueAll	handleQueueSyncAssets	SQL offline check + queues per-asset disk checks
LibrarySyncAssets	handleSyncAssets	Checks a batch of existing assets for offline/online/update
LibraryRemoveAsset	handleAssetRemoval	Removes assets by path (from file watcher unlink event)
LibraryDelete	handleDeleteLibrary	Deletes all assets in a library and removes the library record
LibraryDeleteCheck	handleQueueCleanup	Finds libraries stuck in soft-deleted state and re-queues their deletion

Sources: server/src/services/library.service.ts205-219 server/src/services/library.service.ts238-277 server/src/services/library.service.ts449-475 server/src/services/library.service.ts477-571 server/src/services/library.service.ts612-680 server/src/services/library.service.ts682-694 server/src/services/library.service.ts696-772

---

Concurrency and Locking

DatabaseLock.Library is acquired via databaseRepository.tryLock() at startup. Only the microservice instance that holds this advisory lock will:

• Register and manage the CronJob.LibraryScan cron entry
• Start filesystem watchers

This prevents duplicate cron registrations and multiple watcher instances when more than one microservice process is running.

Sources: server/src/services/library.service.ts42-57

---

Relationship to Asset Processing

After new assets are created by handleSyncFiles, the standard pipeline described in Asset Processing Pipeline is invoked via queuePostSyncJobs(), which queues SidecarCheck → AssetExtractMetadata → AssetGenerateThumbnails → SmartSearch / AssetDetectFaces / AssetEncodeVideo.

Similarly, when handleSyncAssets determines an asset has been modified (mtime changed), it calls queuePostSyncJobs() for that asset, triggering re-extraction of metadata and regeneration of derived files.

Sources: server/src/services/library.service.ts418-428 server/src/services/job.service.ts69-72