This page documents the Transformers library's package structure, build configuration, dependency management, import system, and model file generation tooling. It covers how the library is organized on disk, how dependencies are declared and managed, how the lazy loading mechanism enables efficient imports, and how the modular model converter generates full modeling files from modular source definitions.
For information about the Auto classes and model discovery system, see page 2.1. For details on testing infrastructure, see page 7.3. For the full dependency version workflow, see page 7.2.
The Transformers package is designed to support PyTorch as its primary backend alongside numerous optional dependencies (vision, audio, quantization, etc.) while maintaining a clean import structure. This is achieved through:
setup.py that defines all dependencies and "extras"modeling_*.py files from modular_*.py source filesThe library is organized under src/transformers/ with the following key components:
src/transformers/
├── __init__.py # Main package entry point with lazy loading
├── configuration_utils.py # Configuration base classes
├── modeling_utils.py # PreTrainedModel base class
├── tokenization_utils*.py # Tokenization infrastructure
├── processing_utils.py # Processor infrastructure
├── dependency_versions_table.py # Auto-generated dependency versions
├── models/
│ ├── __init__.py # Models package with lazy loading
│ ├── auto/ # Auto classes for model discovery
│ │ ├── modeling_auto.py
│ │ ├── tokenization_auto.py
│ │ ├── configuration_auto.py
│ │ └── ...
│ ├── bert/ # Individual model implementations
│ ├── gpt2/
│ ├── llama/
│ └── ...
├── utils/
│ ├── dummy_pt_objects.py # PyTorch dummy objects
│ ├── dummy_vision_objects.py # Vision dummy objects
│ ├── import_utils.py # Import helpers
│ └── ...
└── ...
The structure separates:
models/models/auto/utils/Sources: src/transformers/__init__.py1-1157 src/transformers/models/__init__.py1-424
The build system is centralized in setup.py, which defines the package metadata, dependencies, and installation configuration:
Core Dependencies (install_requires):
The library has minimal hard dependencies to reduce conflicts setup.py264-274:
| Dependency | Purpose |
|---|---|
huggingface-hub>=1.3.0,<2.0 | Model hub integration |
numpy>=1.17 | Numerical operations |
packaging>=20.0 | Version comparison utilities |
pyyaml>=5.1 | Configuration file parsing |
regex!=2019.12.17 | Text processing (OpenAI GPT) |
tokenizers>=0.22.0,<=0.23.0 | Fast tokenization |
typer | CLI utilities |
safetensors>=0.4.3 | Safe model serialization |
tqdm>=4.27 | Progress bars |
Sources: setup.py42-357 setup.py264-274
All dependencies are defined in a single _deps list with version constraints setup.py72-165:
This list is processed into a lookup dictionary where keys are package names and values are full version specifications:
DepsTableUpdateCommand is a custom distutils command registered as deps_table_update in setup.py setup.py277-313 When run via make fix-repo (which calls python setup.py deps_table_update), it regenerates src/transformers/dependency_versions_table.py:
The command only runs on the minimum supported Python version (currently 3.10) to ensure a deterministic output setup.py296-298 The generated table is imported at runtime by dependency_versions_check.py to validate installed package versions.
Update workflow:
_deps list in setup.pymake fix-repo to regenerate dependency_versions_table.pySources: setup.py72-169 setup.py277-313 src/transformers/dependency_versions_table.py1-95
Transformers uses "extras" to group optional dependencies by feature setup.py176-261:
Key Extras:
| Extra | Dependencies | Use Case |
|---|---|---|
torch | torch>=2.4, accelerate>=1.1.0 | PyTorch backend |
vision | torchvision, Pillow>=10.0.1,<=15.0 | Image processing |
audio | torchaudio, librosa, pyctcdecode | Audio processing |
sentencepiece | sentencepiece>=0.1.91, protobuf | SentencePiece tokenization |
deepspeed | deepspeed>=0.9.3, accelerate | Distributed training |
testing | pytest, evaluate, test deps | Running tests |
all | Combines torch, vision, audio, video, etc. | Full installation |
dev | all + testing + additional tools | Development |
Installation Examples:
Python Version Constraints: Some extras are conditionally included based on Python version setup.py180-208:
kenlm: Excluded for Python 3.13+mistral-common[image]: Excluded for Python 3.14+ray[tune]: Excluded for Python 3.14+sudachipy / sudachidict_core: Excluded for Python 3.14+Sources: setup.py176-261
pyproject.toml does not define build metadata (that stays in setup.py) but configures the code quality tooling used across the project.
ruff is configured to target Python 3.10 with a line length of 119 pyproject.toml16-36 Selected rule sets include:
| Rule Set | Code | Purpose |
|---|---|---|
| pycodestyle errors | E | Style errors |
| Pyflakes | F | Undefined names, unused imports |
| isort | I | Import ordering |
| pyupgrade | UP | Modern Python syntax |
| refurb | FURB | Idiomatic rewrites |
| flake8-simplify | SIM | Simplification hints |
| flake8-comprehensions | C4 | Comprehension improvements |
E501 (line too long) is explicitly ignored in the lint configuration since the formatter handles line length separately pyproject.toml37-43
ty is listed as a quality dependency and used for type checking in make quality. Its version is pinned in _deps (ty==0.0.12) setup.py127
[tool.coverage.run] in pyproject.toml sets the coverage source to the transformers package and excludes conversion scripts and __main__.py modules pyproject.toml1-14
| Target | Command | Purpose |
|---|---|---|
make style | ruff check --fix, ruff format, custom_init_isort.py, sort_auto_mappings.py | Apply formatting and import sorting |
make quality | ruff check, ruff format --check, ty check | Check without modifying |
make fix-repo | deps_table_update, check_copies, etc. | Regenerate autogenerated files |
make check-repo | python utils/check_repo.py | Validate repository consistency |
Sources: pyproject.toml1-50 Makefile1-60 setup.py127
The lazy loading system uses an _import_structure dictionary that maps module names to lists of exported symbols src/transformers/__init__.py63-266:
Imports are conditionally added based on available backends src/transformers/__init__.py269-466:
Example: PyTorch Backend src/transformers/__init__.py342-466:
The actual lazy loading is implemented by _LazyModule which overrides __getattr__ to import symbols on first access src/transformers/__init__.py57:
When you write from transformers import BertModel, the _LazyModule intercepts the import and dynamically loads the bert model only at that point.
Sources: src/transformers/__init__.py15-57 src/transformers/__init__.py63-466
When optional dependencies are missing, the library provides dummy objects that give clear error messages instead of cryptic import failures.
Each backend has a corresponding dummy objects file src/transformers/utils/dummy_pt_objects.py1-500:
| Module | Backends | Purpose |
|---|---|---|
dummy_pt_objects.py | ["torch"] | PyTorch models and training |
dummy_vision_objects.py | ["vision"] | Image processing |
dummy_tokenizers_objects.py | ["tokenizers"] | Fast tokenizers |
dummy_sentencepiece_objects.py | ["sentencepiece"] | SentencePiece tokenization |
dummy_torchvision_objects.py | ["torchvision"] | Video processing |
dummy_mistral_common_objects.py | ["mistral_common"] | Mistral Common backend |
Example Error:
Sources: src/transformers/utils/dummy_pt_objects.py1-500 src/transformers/__init__.py269-340
The main __init__.py exports approximately 1000+ symbols organized into categories src/transformers/__init__.py63-266:
For type checking, all imports are direct (not lazy) src/transformers/__init__.py469-1157:
This ensures:
Sources: src/transformers/__init__.py469-1157
The models package uses the same lazy loading pattern src/transformers/models/__init__.py14-20 At type-checking time, each model sub-package is imported with a wildcard from .bert import *. At runtime, the define_import_structure() helper from utils.import_utils scans the file to build _import_structure automatically, and _LazyModule is installed into sys.modules:
The define_import_structure() function reads the TYPE_CHECKING block of the calling file to discover what sub-modules and names should be lazily loaded.
Each model follows a consistent structure:
src/transformers/models/bert/
├── __init__.py # Model package exports
├── configuration_bert.py # BertConfig
├── modeling_bert.py # BertModel, BertForXXX
├── tokenization_bert.py # BertTokenizer (slow)
├── tokenization_bert_fast.py # BertTokenizerFast
└── convert_bert_original_tf_checkpoint.py # Optional conversion script
Sources: src/transformers/models/__init__.py1-20
The modular model converter is a code generation system that allows new model implementations to be written as short modular_<model>.py files that inherit from and override existing model classes. A converter script then expands these into complete standalone modeling_<model>.py files that contain no dynamic inheritance. This separation keeps model contributions concise while ensuring each committed file is self-contained and readable.
Key files:
| File | Role |
|---|---|
utils/modular_model_converter.py | Main converter — reads modular_*.py, writes modeling_*.py |
utils/check_modular_conversion.py | CI check — verifies committed files match converter output |
utils/create_dependency_mapping.py | Builds a topological ordering of modular files for processing |
A modular_<model>.py file (e.g., src/transformers/models/gemma/modular_gemma.py) imports from existing model modules and subclasses only what needs to change. For example:
The converter resolves the full inheritance chain and inlines all inherited code into the output modeling_gemma.py, which then contains no cross-model imports.
Generated files carry a warning header preventing manual edits utils/modular_model_converter.py44-50:
# 🚨 This file was automatically generated from modular_gemma.py.
# Do NOT edit this file manually as any edits will be overwritten ...
Modular model file to modeling file conversion pipeline:
Sources: utils/modular_model_converter.py1-60 utils/create_dependency_mapping.py1-50
| Symbol | File | Role |
|---|---|---|
convert_modular_file | modular_model_converter.py | Entry point for a single file conversion |
run_ruff | modular_model_converter.py | Formats output with ruff after generation |
find_priority_list | create_dependency_mapping.py | Topological sort of modular files by dependency |
AbsoluteImportTransformer | modular_integrations.py | Rewrites relative imports to absolute in output |
RelativeImportTransformer | modular_integrations.py | Inverse transform for internal use |
preserve_case_replace | modular_model_converter.py | Name substitution preserving case during class renaming |
AUTO_GENERATED_MESSAGE | modular_model_converter.py | Warning banner placed at top of every generated file |
Sources: utils/modular_model_converter.py44-62 utils/check_modular_conversion.py1-30
utils/check_modular_conversion.py is run as part of the make fix-repo flow and in CI. It:
convert_modular_file on each modular_*.py to produce a fresh outputmodeling_*.pyThis ensures the committed modeling files always reflect the current modular sources. The BACKUP_EXT = ".modular_backup" constant is used as a scratch extension during comparison utils/check_modular_conversion.py23
Sources: utils/check_modular_conversion.py1-50 Makefile1-40
The package provides a CLI through setuptools entry points setup.py342:
After installation, the transformers command is available:
Sources: setup.py342
The package includes non-Python files for CUDA kernels and C++ extensions setup.py339:
This ensures:
.cu, .cuh).cpp, .h).pyx)py.typed)Sources: setup.py338-340
The repository includes automated consistency checks utils/check_repo.py1-2000:
Checks performed:
models/__init__.py__init__.pyUsage:
Sources: utils/check_repo.py1-2000 Makefile20-26
The Transformers package structure is designed for:
Key Files:
This architecture enables the library to support 200+ models, multiple backends, and numerous optional features while maintaining fast import times and clear error messages.
Refresh this wiki
This wiki was recently refreshed. Please wait 2 days to refresh again.