Hardware Context System

Relevant source files

• doc/codecs.texi
• doc/formats.texi
• doc/multithreading.txt
• libavcodec/avcodec.c
• libavcodec/avcodec.h
• libavcodec/avcodec_internal.h
• libavcodec/codec_internal.h
• libavcodec/codec_par.c
• libavcodec/codec_par.h
• libavcodec/decode.c
• libavcodec/decode.h
• libavcodec/lcevcdec.c
• libavcodec/lcevcdec.h
• libavcodec/libjxldec.c
• libavcodec/libjxlenc.c
• libavcodec/options.c
• libavcodec/options_table.h
• libavcodec/packet.c
• libavcodec/packet.h
• libavcodec/packet_internal.h
• libavcodec/progressframe.h
• libavcodec/pthread.c
• libavcodec/pthread_frame.c
• libavcodec/pthread_slice.c
• libavcodec/tests/avcodec.c
• libavcodec/thread.h
• libavcodec/vdpau.c
• libavcodec/vdpau.h
• libavcodec/version_major.h
• libavcodec/vulkan_av1.c
• libavcodec/vulkan_decode.c
• libavcodec/vulkan_decode.h
• libavcodec/vulkan_h264.c
• libavcodec/vulkan_hevc.c
• libavcodec/vulkan_video.c
• libavcodec/vulkan_video.h
• libavcodec/vulkan_vp9.c
• libavcodec/webp.c
• libavfilter/vf_lcevc.c
• libavfilter/vf_nlmeans_vulkan.c
• libavfilter/vulkan_filter.c
• libavformat/options_table.h
• libavutil/hwcontext_vulkan.c
• libavutil/hwcontext_vulkan.h
• libavutil/vulkan.c
• libavutil/vulkan.h
• libavutil/vulkan_functions.h
• libavutil/vulkan_loader.h

Purpose and Scope

The Hardware Context System provides a unified abstraction layer for managing hardware-accelerated processing in FFmpeg. The primary public interface is libavutil/hwcontext.h, which exposes AVHWDeviceContext and AVHWFramesContext — backend-agnostic types used to represent a hardware device and a pool of hardware frames respectively. Each supported backend (CUDA, VAAPI, Vulkan, etc.) extends these with backend-specific context structures defined in separate headers such as libavutil/hwcontext_vulkan.h.

This document focuses on:

• The general backend-agnostic public API (hwcontext.h)
• Vulkan-specific structures and implementation details as the primary cross-platform backend
• How decoders and filters create, share, and use hardware contexts

For information about AVFrame and buffer management, see page 3.1.1. For Vulkan-specific codec and filter implementations, see page 7.1.

Sources: libavutil/hwcontext_vulkan.h1-50 libavutil/hwcontext_vulkan.c1-74

Architecture Overview

Two-Level Context Hierarchy

The hardware context system uses a two-level hierarchy:

1. Device Context (AVHWDeviceContext): Represents a hardware device and its capabilities. Contains the Vulkan device, queue families, extensions, and features.

2. Frames Context (AVHWFramesContext): Represents a pool of frames with specific format and dimensions. Contains image format information and execution pools for operations.

Sources: libavutil/hwcontext_vulkan.h59-275 libavutil/hwcontext_vulkan.c123-171 libavutil/hwcontext_vulkan.c173-194

---

General Public API

The general API is declared in libavutil/hwcontext.h and is backend-agnostic. Callers interact with AVHWDeviceContext and AVHWFramesContext using the av_hw* functions regardless of the underlying hardware type.

AVHWDeviceType — Supported Backends

The AVHWDeviceType enum identifies the hardware backend. Each type has a corresponding hwcontext_<name>.h header that defines the backend-specific public context structure (e.g., AVVulkanDeviceContext in hwcontext_vulkan.h).

Enum Value	Backend Header	Platform
AV_HWDEVICE_TYPE_CUDA	hwcontext_cuda.h	Linux, Windows
AV_HWDEVICE_TYPE_DRM	hwcontext_drm.h	Linux
AV_HWDEVICE_TYPE_DXVA2	hwcontext_dxva2.h	Windows
AV_HWDEVICE_TYPE_D3D11VA	hwcontext_d3d11va.h	Windows
AV_HWDEVICE_TYPE_D3D12VA	hwcontext_d3d12va.h	Windows
AV_HWDEVICE_TYPE_VAAPI	hwcontext_vaapi.h	Linux
AV_HWDEVICE_TYPE_VDPAU	hwcontext_vdpau.h	Linux
AV_HWDEVICE_TYPE_OPENCL	hwcontext_opencl.h	Cross-platform
AV_HWDEVICE_TYPE_QSV	hwcontext_qsv.h	Linux, Windows
AV_HWDEVICE_TYPE_MEDIACODEC	hwcontext_mediacodec.h	Android
AV_HWDEVICE_TYPE_VULKAN	hwcontext_vulkan.h	Cross-platform

Use av_hwdevice_find_type_by_name() to map a name string (e.g. "vulkan", "cuda") to a type value, and av_hwdevice_get_type_name() to do the reverse.

Public API Functions

Device context functions:

Function	Purpose
av_hwdevice_ctx_alloc(type)	Allocate an AVHWDeviceContext of the given type; caller fills in backend fields
av_hwdevice_ctx_init(ref)	Initialize a manually-filled device context
av_hwdevice_ctx_create(ref, type, device, opts, flags)	Allocate and open a device in one call; most common entry point
av_hwdevice_ctx_create_derived(dst, type, src, flags)	Create a context of a new type interoperating with an existing context (e.g., VAAPI from DRM)

Frames context functions:

Function	Purpose
av_hwframes_ctx_alloc(device_ref)	Allocate an AVHWFramesContext linked to a device context
av_hwframes_ctx_init(ref)	Initialize the frames context (creates the frame pool)
av_hwframe_get_buffer(frames_ref, frame, flags)	Allocate a hardware frame from the pool
av_hwframe_transfer_data(dst, src, flags)	Transfer data between hardware and software frames
av_hwframe_map(dst, src, flags)	Map a hardware frame into a different address space (e.g., host-visible)
av_hwframe_transfer_get_formats(frames_ref, dir, formats, flags)	Query the pixel formats available for av_hwframe_transfer_data

Context Lifecycle Diagram

Title: AVHWDeviceContext and AVHWFramesContext lifecycle

Sources: libavutil/hwcontext_vulkan.h59-275

Device Context Structure

AVVulkanDeviceContext

The AVVulkanDeviceContext structure defines the public API for device management. Key fields:

Field	Type	Purpose
inst	VkInstance	Vulkan instance (required version 1.3+)
phys_dev	VkPhysicalDevice	Physical device handle
act_dev	VkDevice	Logical device handle
qf	AVVulkanDeviceQueueFamily*	Array of queue families
nb_qf	int	Number of queue families
enabled_inst_extensions	const char**	Enabled instance extensions
enabled_dev_extensions	const char**	Enabled device extensions
device_features	VkPhysicalDeviceFeatures2	Enabled device features

Sources: libavutil/hwcontext_vulkan.h59-208

VulkanDevicePriv - Internal State

The VulkanDevicePriv structure contains internal state not exposed through the public API:

• FFVulkanContext: Core Vulkan context with function pointers and extensions
• Properties: Device properties, memory properties, driver information
• Features: Comprehensive feature flags including Vulkan 1.1/1.2/1.3 features
• Queue Management: Mutexes for thread-safe queue access
• Settings: Configuration options like use_linear_images, contiguous_planes

Sources: libavutil/hwcontext_vulkan.c123-171

Queue Family Management

Queue families are managed through AVVulkanDeviceQueueFamily structures. The system supports:

• Graphics queues: Rendering operations
• Compute queues: Compute shader execution
• Transfer queues: Memory transfer operations
• Video decode/encode queues: Hardware video acceleration

The function ff_vk_qf_find() at libavutil/vulkan.c288-299 locates appropriate queue families based on required capabilities.

Sources: libavutil/hwcontext_vulkan.h33-45 libavutil/vulkan.c288-299

Frames Context Structure

AVVulkanFramesContext

The AVVulkanFramesContext defines frame pool properties:

Field	Type	Purpose
format[3]	VkFormat	Vulkan formats for each plane (up to 3 planes)
tiling	VkImageTiling	VK_IMAGE_TILING_OPTIMAL or LINEAR
usage	VkImageUsageFlags	Image usage flags (SAMPLED, STORAGE, TRANSFER, VIDEO_DECODE)
create_pnext	void*	Extended creation info (DRM modifiers, etc.)
alloc_pnext	void*	Extended allocation info (external memory)
img_flags	VkImageCreateFlags	Image creation flags

Sources: libavutil/hwcontext_vulkan.h244-275

VulkanFramesPriv - Execution Pools

VulkanFramesPriv maintains execution pools for different operation types:

• compute_exec: Image format conversions and processing
• upload_exec: CPU-to-GPU data transfers
• download_exec: GPU-to-CPU data transfers

Each pool contains multiple FFVkExecContext instances for parallel command buffer recording.

Sources: libavutil/hwcontext_vulkan.c173-194 libavutil/vulkan.c301-364

Memory Management

Memory Allocation Strategy

Memory allocation follows this pattern:

1. Query Requirements: vkGetImageMemoryRequirements2() determines size and alignment
2. Find Memory Type: ff_vk_alloc_mem() at libavutil/hwcontext_vulkan.c1033-1131 finds compatible memory type
3. Allocate: vkAllocateMemory() allocates device memory
4. Bind: vkBindImageMemory2() binds memory to image

Memory type selection prioritizes:

• DEVICE_LOCAL for GPU operations
• HOST_VISIBLE + HOST_COHERENT for staging buffers
• HOST_CACHED when available for read operations

Sources: libavutil/hwcontext_vulkan.c1033-1131 libavutil/hwcontext_vulkan.c1174-1299

Buffer Management

FFVkBuffer at libavutil/vulkan.h125-143 encapsulates Vulkan buffer management:

Field	Purpose
buf	VkBuffer handle
mem	Associated VkDeviceMemory
mapped_mem	CPU-side pointer when mapped
address	Device address for shader access
virtual_offset	Offset for host-mapped buffers

Key buffer functions:

• ff_vk_create_buf(): Create and allocate buffer
• ff_vk_map_buffer(): Map to CPU address space
• ff_vk_flush_buffer(): Flush CPU writes to device
• ff_vk_get_pooled_buffer(): Get buffer from thread-safe pool

Sources: libavutil/vulkan.h125-143 libavutil/vulkan.c1167-1341

Data Transfer Operations

Upload and Download Paths

Transfer operations use staging buffers for CPU-GPU communication:

Upload (vulkan_transfer_data_from() at libavutil/hwcontext_vulkan.c2761-3031):

1. Allocate staging buffer with HOST_VISIBLE memory
2. Copy CPU data to staging buffer via memcpy()
3. Record vkCmdCopyBufferToImage() command
4. Submit command buffer with appropriate barriers

Download (vulkan_transfer_data_to() at libavutil/hwcontext_vulkan.c3033-3327):

1. Allocate staging buffer with HOST_VISIBLE memory
2. Record vkCmdCopyImageToBuffer() command
3. Submit command buffer and wait for completion
4. Copy staging buffer to CPU data via memcpy()

Sources: libavutil/hwcontext_vulkan.c2761-3327

Host Image Copy Extension

When VK_EXT_host_image_copy is available, the system can perform direct CPU-GPU transfers without staging buffers. This is checked at libavutil/hwcontext_vulkan.c2794-2828 and uses vkCopyMemoryToImageEXT() for uploads and vkCopyImageToMemoryEXT() for downloads.

Sources: libavutil/hwcontext_vulkan.c2794-2828 libavutil/hwcontext_vulkan.c3073-3147

External Memory Interoperability

DMA-BUF Import/Export

DMA-BUF support enables zero-copy sharing with other APIs:

Export (Vulkan to DMA-BUF):

• Check FF_VK_EXT_EXTERNAL_DMABUF_MEMORY flag
• Use vkGetMemoryFdKHR() to get file descriptor
• FD can be imported by VAAPI, V4L2, or other APIs

Import (DMA-BUF to Vulkan):

• Create VkImportMemoryFdInfoKHR with external FD
• Pass as pNext to VkMemoryAllocateInfo
• Memory is imported without copying

DRM Modifiers: When FF_VK_EXT_DRM_MODIFIER_FLAGS is enabled, images can specify format modifiers for optimal hardware layout compatibility.

Sources: libavutil/hwcontext_vulkan.c1301-1473 libavutil/hwcontext_vulkan.c1992-2253

CUDA Interoperability

AVVkFrameInternal at libavutil/hwcontext_vulkan.c196-212 caches CUDA imports for performance:

The CUDA import is expensive, so memory and semaphores are kept imported throughout the frame's lifetime.

Sources: libavutil/hwcontext_vulkan.c196-212 libavutil/hwcontext_vulkan.c2373-2696

Host Memory Mapping

The function ff_vk_host_map_buffer() at libavutil/vulkan.c1343-1410 can map system RAM directly as a Vulkan buffer without copying:

1. Query host pointer properties with vkGetMemoryHostPointerPropertiesEXT()
2. Create VkImportMemoryHostPointerInfoEXT with host pointer
3. Allocate memory with import info in pNext chain
4. Create buffer and bind imported memory

This requires FF_VK_EXT_EXTERNAL_HOST_MEMORY extension and proper alignment.

Sources: libavutil/vulkan.c1343-1410 libavutil/hwcontext_vulkan.c1033-1131

Internal Implementation Details

FFVulkanContext

FFVulkanContext at libavutil/vulkan.h313-365 is the internal context used throughout Vulkan operations:

Key fields:

• vkfn: All Vulkan function pointers loaded via ff_vk_load_functions()
• extensions: Bitmask of enabled extensions (FFVulkanExtensions)
• props, mprops, feats: Device capabilities
• qfs[]: Array of enabled queue family indices

Initialization: ff_vk_load_props() at libavutil/vulkan.c147-286 queries all device properties and capabilities.

Sources: libavutil/vulkan.h313-365 libavutil/vulkan.c147-286

Execution Pool System

The execution pool system provides thread-safe command buffer management:

FFVkExecPool (ff_vk_exec_pool_init() at libavutil/vulkan.c366-527):

• Array of FFVkExecContext instances (typically 2-4 per pool)
• Each context has its own command pool and command buffer
• Atomic index for round-robin context selection
• Optional query pool for performance queries

FFVkExecContext:

• Manages command buffer lifecycle
• Tracks buffer, frame, and semaphore dependencies
• Ensures proper resource lifetime through reference counting

Typical usage:

Sources: libavutil/vulkan.h291-311 libavutil/vulkan.h145-208 libavutil/vulkan.c366-627

Format Conversion Tables

The format system maintains a comprehensive mapping table at libavutil/hwcontext_vulkan.c402-519:

Format entry structure:

• vkf: Primary Vulkan format (e.g., VK_FORMAT_G8_B8R8_2PLANE_420_UNORM)
• pixfmt: Corresponding AVPixelFormat (e.g., AV_PIX_FMT_NV12)
• aspect: Image aspect flags for plane selection
• vk_planes: Number of Vulkan planes in native format
• nb_images: Number of separate images to create
• fallback[]: Alternative formats if native format unsupported

Format selection (vkfmt_from_pixfmt2() at libavutil/hwcontext_vulkan.c538-629):

1. Look up format entry by pixel format
2. Query device support for native format
3. Fall back to per-plane formats if needed
4. Check storage image support if required

Sources: libavutil/hwcontext_vulkan.c402-629

Feature and Extension Management

Extension management uses a 64-bit bitmask for efficient checking:

Extension flags (defined in libavutil/vulkan_functions.h29-75):

• Each extension has a unique bit flag
• Special flags: FF_VK_EXT_NO_FLAG (always enabled), FF_VK_EXT_PORTABILITY_SUBSET
• Video extensions: FF_VK_EXT_VIDEO_DECODE_H264, FF_VK_EXT_VIDEO_DECODE_HEVC, etc.

Optional extensions (libavutil/hwcontext_vulkan.c686-758):

• System probes available extensions at initialization
• Enables supported extensions automatically
• Extensions can be manually specified in AVVulkanDeviceContext

Feature management (libavutil/hwcontext_vulkan.c75-121 libavutil/hwcontext_vulkan.c215-288):

• VulkanDeviceFeatures structure chains all feature structures
• Features are queried during initialization
• Only needed features are enabled during device creation

Sources: libavutil/vulkan_functions.h29-75 libavutil/hwcontext_vulkan.c75-288 libavutil/hwcontext_vulkan.c686-758