Service Lifecycle Management

Relevant source files

• .changes/added/3131.md
• bin/fuel-core/Cargo.toml
• bin/fuel-core/src/cli.rs
• bin/fuel-core/src/cli/archive.rs
• bin/fuel-core/src/cli/run.rs
• bin/fuel-core/src/cli/run/graphql.rs
• bin/fuel-core/src/cli/snapshot.rs
• crates/fuel-core/Cargo.toml
• crates/fuel-core/src/graphql_api.rs
• crates/fuel-core/src/graphql_api/api_service.rs
• crates/fuel-core/src/service.rs
• crates/fuel-core/src/service/adapters.rs
• crates/fuel-core/src/service/config.rs
• crates/fuel-core/src/service/sub_services.rs
• crates/services/src/service.rs
• crates/services/src/state.rs
• crates/services/sync/src/service.rs
• tests/tests/dos.rs

This document describes the service lifecycle management system in fuel-core, which provides a standardized pattern for starting, stopping, and monitoring asynchronous services throughout the node. This infrastructure ensures orderly initialization sequences, graceful shutdown, and error recovery.

For information about the overall service orchestration and sub-service composition, see FuelService Architecture. For details on individual sub-services, see Sub-Services Overview.

---

Overview

The service lifecycle system is built around three core abstractions:

1. State Machine: A six-state lifecycle model tracking service status from initialization through shutdown
2. ServiceRunner Pattern: A wrapper that manages service execution and provides control operations
3. RunnableService/RunnableTask Traits: Interfaces that services implement to integrate with the lifecycle system

All services in fuel-core follow this pattern, from the top-level FuelService down to individual sub-services like TxPool, GraphQL, and BlockProducer.

Sources: crates/services/src/service.rs1-52 crates/fuel-core/src/service.rs1-76

---

Service State Machine

State Definitions

Services transition through a well-defined state machine represented by the State enum:

State Descriptions:

State	Description	Duration
NotStarted	Service created but not yet started	Until start() called
Starting	Service initializing via into_task()	Until initialization completes
Started	Service running normally, executing run() loop	Until stop() called or error
Stopping	Service shutting down gracefully	Until shutdown() completes
Stopped	Service cleanly terminated	Terminal state
StoppedWithError(String)	Service terminated due to panic	Terminal state

Sources: crates/services/src/state.rs5-47

State Transitions

The state machine enforces valid transitions. Invalid operations return errors:

• Calling start() on a service already starting/started returns an error
• Calling stop() on an already stopped service returns false
• Only NotStarted, Starting, and Started states can transition to Stopping

The state is managed via a tokio::sync::watch channel, allowing multiple observers to monitor state changes reactively.

Sources: crates/services/src/service.rs261-321

---

StateWatcher

The StateWatcher provides reactive monitoring of service state changes. It wraps a watch::Receiver<State> and adds utility methods:

Key Methods:

Method	Purpose
borrow()	Get current state without marking as seen
borrow_and_update()	Get current state and mark as seen
changed().await	Wait for state change
while_started().await	Loop until state is not Started
wait_stopping_or_stopped().await	Wait until service begins or completes shutdown

The while_started() method is particularly important for task loops - it continues until the service receives a stop signal.

Sources: crates/services/src/state.rs49-136

---

ServiceRunner Pattern

Structure

The ServiceRunner<S> type wraps a service and manages its lifecycle:

The ServiceRunner separates control operations (start/stop) from service execution. When created, it spawns a background task that:

1. Waits for the Starting state signal
2. Calls RunnableService::into_task() to initialize
3. Transitions to Started state
4. Repeatedly calls RunnableTask::run() while state is Started
5. Calls RunnableTask::shutdown() when stopping
6. Catches any panics and transitions to StoppedWithError

Sources: crates/services/src/service.rs188-253 crates/services/src/service.rs323-379

Control Operations

The Service trait defines the control interface:

Operation	Behavior	Return
start()	Send start signal	Result<(), anyhow::Error>
start_and_await()	Start and wait until Started or stopped	Result<State>
stop()	Send stop signal	bool (true if was running)
stop_and_await()	Stop and wait for Stopped	Result<State>
state()	Get current state	State
state_watcher()	Get StateWatcher for monitoring	StateWatcher

Sources: crates/services/src/service.rs22-52 crates/services/src/service.rs256-321

Creation and Initialization

Services are created using the new or new_with_params constructors:

The shared_data() method returns a cloneable shared state that external code can use to interact with the service. For example, TxPool exposes methods to insert transactions via its shared state.

Sources: crates/services/src/service.rs209-230 crates/services/src/service.rs381-424

---

RunnableService and RunnableTask Traits

RunnableService Trait

Services implement RunnableService to define initialization logic:

Associated Types:

• SharedData: Cloneable state exposed to external callers (e.g., TxPoolSharedState)
• Task: The runnable task type returned by into_task()
• TaskParams: Optional parameters passed during initialization

The into_task() method is called when the service starts. It can perform setup operations like connecting to peers or initializing databases. The state_watcher parameter allows monitoring for early shutdown signals.

Sources: crates/services/src/service.rs54-84

RunnableTask Trait

After initialization, the RunnableTask trait defines the execution loop:

TaskNextAction Enum:

The run() method contains one iteration of the service's main logic. It should:

• Check watcher.borrow() periodically to detect stop signals
• Return quickly (not block indefinitely)
• Return TaskNextAction::Continue to run again
• Return TaskNextAction::Stop to initiate shutdown

The shutdown() method performs cleanup when the service stops.

Sources: crates/services/src/service.rs86-186

Example: GraphQL Service

The GraphQL API service demonstrates this pattern:

The GraphqlService::into_task() method:

1. Creates an AsyncProcessor with the specified thread count
2. Binds an axum::Server to the TCP listener
3. Configures graceful shutdown triggered by state changes
4. Returns a Task containing the server future

The Task::run() method simply awaits the server future, which internally loops handling HTTP requests.

Sources: crates/fuel-core/src/graphql_api/api_service.rs161-233

---

FuelService Orchestration

Structure

The FuelService is the top-level orchestrator that manages all sub-services:

The FuelService::new() constructor:

1. Validates and makes configuration consistent
2. Checks database version compatibility
3. Calls init_sub_services() to create all sub-services
4. Wraps them in a Task and ServiceRunner

Sources: crates/fuel-core/src/service.rs138-197

Sub-Service Initialization

The init_sub_services() function in sub_services.rs creates all services in a specific order:

Critical Ordering:

1. Gas Price Service - Must start first to provide gas price data
2. TxPool - Depends on gas price service
3. Chain State Info - Tracks block heights
4. Relayer (if enabled) - Syncs DA layer events
5. P2P, Sync (if enabled) - Network services
6. GraphQL - API endpoint
7. GraphQL Worker - Off-chain indexing
8. TxStatus Manager - Transaction status tracking
9. Block Producer (PoA) - MUST BE LAST to ensure all dependencies are ready

The block producer must be last because it depends on all other services being available.

Sources: crates/fuel-core/src/service/sub_services.rs133-592

Startup Sequence

The FuelService::start_and_await() method orchestrates the startup:

The into_task() method starts each sub-service sequentially, using tokio::select! to respect early stop signals. After all services start, it sends the block_production_ready_signal, which unblocks the PoA service to begin producing blocks.

Sources: crates/fuel-core/src/service.rs415-421 crates/fuel-core/src/service.rs478-511

Shutdown Sequence

The Task::run() method monitors all sub-services for shutdown:

The shutdown sequence is defensive:

• If any sub-service stops (intentionally or due to error), the entire FuelService stops
• Each sub-service is stopped explicitly, with errors logged but not propagated
• The database is shut down last to ensure all data is persisted

Sources: crates/fuel-core/src/service.rs513-547

---

Error Handling and Panic Recovery

Panic Catching

The service infrastructure catches panics at two levels:

1. Service Initialization: Panics in into_task() are not caught - the service fails to start
2. Service Execution: Panics in run() or shutdown() are caught and converted to StoppedWithError

When a panic occurs:

1. The panic is caught by catch_unwind()
2. Panic information is extracted with panic_to_string()
3. State transitions to StoppedWithError(panic_info)
4. After notifying watchers, the panic is resumed with resume_unwind()

This ensures graceful notification of other services while still propagating the panic for visibility.

Sources: crates/services/src/service.rs336-378 crates/services/src/service.rs426-444

TaskNextAction Error Handling

The TaskNextAction::ErrorContinue variant allows services to report errors without stopping:

This is useful for transient errors like network timeouts. The error is logged at the error level, but the service continues running.

Helper Macros:

Sources: crates/services/src/service.rs86-145

Sub-Service Error Propagation

When a sub-service stops with an error, the FuelService detects it:

The continue_on_error configuration flag controls whether some errors are tolerated or cause immediate shutdown. This is primarily used during development.

Sources: crates/fuel-core/src/service.rs513-531

---

Service Configuration and Initialization

CLI Entry Point

The fuel-core run command is the standard entry point:

The exec() function:

1. Creates the service with a ShutdownListener for signal handling
2. Starts the service with tokio::select! to handle early cancellation
3. Monitors both the service and OS signals (SIGTERM, SIGINT)
4. Calls send_stop_signal_and_await_shutdown() on termination

Sources: bin/fuel-core/src/cli/run.rs840-866

Configuration Construction

The Command::get_config() method builds the complete configuration:

Configuration validation includes:

• allow_syscall requires debug mode
• Gas price validation (max >= min)
• State rewind policy calculation for RocksDB

The configuration is made consistent with Config::make_config_consistent(), which ensures flags like utxo_validation are synchronized across services.

Sources: bin/fuel-core/src/cli/run.rs342-798 crates/fuel-core/src/service/config.rs302-316

---

Service Metrics and Monitoring

FuturesMetrics

Each service is instrumented with futures metrics:

The metrics track:

• Number of spawned tasks per service
• Task execution durations
• Task queue depths

Sources: crates/services/src/service.rs227-228

State Monitoring in Tests

Test helpers provide utilities for monitoring service state:

This is extensively used in integration tests to verify proper service startup and shutdown.

Sources: crates/fuel-core/src/service.rs628-651

---

Key Patterns and Best Practices

1. Shared State Design

Services expose shared state via cloneable handles:

The shared state typically uses Arc, tokio::sync::watch, or other concurrency primitives to enable safe multi-threaded access.

Sources: crates/fuel-core/src/service/adapters.rs317-324

2. Port-Based Architecture

Services depend on port traits, not concrete implementations:

This enables dependency injection and testability. See Adapters and Ports Pattern for details.

Sources: crates/fuel-core/src/service/adapters.rs447-474

3. Initialization Order Matters

Services must start in dependency order. The init_sub_services() function carefully orders initialization:

Sources: crates/fuel-core/src/service/sub_services.rs549-589

4. Graceful Shutdown

Services should:

• Monitor StateWatcher for stop signals
• Return TaskNextAction::Stop to initiate shutdown
• Clean up resources in shutdown() method
• Use timeouts for operations that might hang

Sources: crates/services/src/service.rs177-186

5. Testing Service Lifecycle

Tests verify that stopping any sub-service triggers full shutdown:

This ensures the orchestration logic correctly handles individual service failures.

Sources: crates/fuel-core/src/service.rs565-626