This page documents the video capture and encoding subsystem of the scrcpy Android server. It explains how the server captures video from either the device display or camera, encodes it using hardware codecs, and streams it to the desktop client.
For information about audio capture and encoding, see page 3.3. For details on how the client receives and decodes video, see page 2.5.
The video capture and encoding pipeline runs entirely on the Android device as part of the scrcpy-server Java process. It consists of three main stages:
The pipeline operates asynchronously with the encoder consuming frames as they become available from the capture source.
The following diagram maps system concepts to the actual Java classes involved.
Diagram: Class-level video pipeline
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceCapture.java1-97 server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java1-60 server/src/main/java/com/genymobile/scrcpy/video/VideoFilter.java1-119
SurfaceCapture (server/src/main/java/com/genymobile/scrcpy/video/SurfaceCapture.java) is the abstract base class for all video sources. SurfaceEncoder calls its methods in a defined lifecycle sequence.
Lifecycle methods:
| Method | Called when |
|---|---|
init(CaptureListener) | Once before first capture; registers the reset listener |
prepare() | Before each capture start; computes videoSize and transform |
start(Surface) | Begins rendering frames to the provided Surface |
stop() | Stops rendering (e.g., releases OpenGLRunner) |
release() | Final cleanup (called once after last capture) |
getSize() | Returns the current Size after prepare() |
setMaxSize(int) | Called by encoder to request downscaling on error |
isClosed() | Signals internal closure (e.g., camera disconnect) |
invalidate() | Protected; subclass calls this to trigger encoder reset |
requestInvalidate() | Called externally (e.g., user request) |
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceCapture.java1-97
The server supports three concrete SurfaceCapture implementations, selected based on command-line options.
ScreenCapture (server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java) mirrors an existing display identified by displayId.
Key behaviors:
DisplaySizeMonitor in init() to detect resolution changes; calls invalidate() when the display size changes, triggering an encoder reset server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java63-65prepare(), builds a VideoFilter from the display's current rotation, any crop rect, orientation lock, and --angle server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java88-101start(), creates a virtual display using DisplayManager.createVirtualDisplay(), falling back to the SurfaceControl API if that fails server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java127-144VideoFilter transform is non-null, inserts an OpenGLRunner + AffineOpenGLFilter between the virtual display surface and the encoder surface server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java115-125VirtualDisplayListener with a PositionMapper so that input events can be correctly translated server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java146-160NewDisplayCapture (server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java) creates a new virtual display rather than mirroring an existing one (activated via --new-display).
Key behaviors:
init(), resolves display size and DPI from the main display if not explicitly specified server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java85-102prepare(), builds two VideoFilter chains — one for the OpenGL transform (displayTransform) and one for input event mapping (eventTransform) — because the virtual display's physical orientation differs from its logical orientation server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java105-167startNew(), creates the virtual display with a set of flags controlling trust level, focus ownership, and system decoration visibility server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java169-207VIRTUAL_DISPLAY_FLAG_TRUSTED, VIRTUAL_DISPLAY_FLAG_OWN_DISPLAY_GROUP, and VIRTUAL_DISPLAY_FLAG_ALWAYS_UNLOCKED server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java183-192CameraCapture (server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java) uses the Camera2 API to stream from a device camera.
Key behaviors:
selectCamera() picks a camera by explicit ID or by CameraFacing (front/back/external) server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java142-170selectSize() queries StreamConfigurationMap for supported output sizes, filtering by maxSize and aspectRatio, then selects the largest matching size server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java173-238VFLIP_MATRIX as the texture transform passed to OpenGLRunner, because the SurfaceTexture transform returned by the Camera2 API often contains an unexpected 90° rotation server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java49-54 server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java256-261CameraConstrainedHighSpeedCaptureSession and createHighSpeedRequestList() server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java408-411isClosed() returns true when the camera disconnects, causing the encoder to terminate cleanly server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java418-420Comparison of capture implementations:
| Property | ScreenCapture | NewDisplayCapture | CameraCapture |
|---|---|---|---|
| Source | Existing display | New virtual display | Camera2 API |
| Android API used | DisplayManager / SurfaceControl | DisplayManager | CameraManager |
| Input events supported | Yes (via PositionMapper) | Yes (via PositionMapper) | No |
| Rotation handling | DisplaySizeMonitor + invalidate() | DisplaySizeMonitor + invalidate() | Fixed VFLIP_MATRIX |
| Display flags | — | Extensive flag set | — |
Sources: server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java1-219 server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java1-269 server/src/main/java/com/genymobile/scrcpy/video/CameraCapture.java1-426
VideoFilter (server/src/main/java/com/genymobile/scrcpy/video/VideoFilter.java) accumulates a chain of 2D affine transforms describing how the raw capture frame should be transformed before encoding. It also tracks the resulting output Size after all transforms are applied.
Transform operations:
| Method | Effect on size | Transform applied |
|---|---|---|
addCrop(Rect, boolean) | Size = crop dimensions | AffineMatrix.reframe(x, y, w, h) |
addRotation(int ccwRotation) | Swaps width/height if rotation is 90°/270° | AffineMatrix.rotateOrtho(ccwRotation) |
addOrientation(int displayRotation, boolean locked, Orientation) | May swap dimensions | Combines reverse-display-rotation + orientation flip/rotate |
addAngle(double cwAngle) | No size change | AffineMatrix.rotate(ccwAngle).withAspectRatio().fromCenter() |
addResize(Size) | Size = targetSize | Sets transform = IDENTITY if null (forces OpenGL pass-through) |
getInverseTransform() returns the inverse of the accumulated matrix. This inverse is used in two places:
AffineOpenGLFilter (OpenGL operates on texture coordinates, which are the inverse of image transforms)PositionMapper (to map client click positions back to device coordinates)Sources: server/src/main/java/com/genymobile/scrcpy/video/VideoFilter.java1-119
SurfaceEncoder (server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java) implements AsyncProcessor and drives the full encode loop on a dedicated "video" thread.
Diagram: SurfaceEncoder encode loop
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java65-147
createFormat() builds the MediaFormat passed to MediaCodec.configure():
| MediaFormat Key | Value / Source |
|---|---|
KEY_MIME | Codec MIME type (e.g., video/avc) |
KEY_BIT_RATE | options.getVideoBitRate() |
KEY_FRAME_RATE | Fixed at 60 (nominal; actual rate is variable) |
KEY_COLOR_FORMAT | COLOR_FormatSurface |
KEY_COLOR_RANGE | COLOR_RANGE_LIMITED (API 24+) |
KEY_I_FRAME_INTERVAL | DEFAULT_I_FRAME_INTERVAL = 10 seconds |
KEY_REPEAT_PREVIOUS_FRAME_AFTER | REPEAT_FRAME_DELAY_US = 100 000 µs |
max-fps-to-encoder | options.getMaxFps() (if > 0) |
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java256-286
| Codec | MIME Type | Notes |
|---|---|---|
| H.264 (AVC) | video/avc | Default; broadest device support |
| H.265 (HEVC) | video/hevc | Better compression |
| AV1 | video/av01 | Best compression; newer devices only |
Codec selection is from options.getVideoCodec() (CLI: --video-codec). Encoder selection uses MediaCodec.createEncoderByType() by default, or MediaCodec.createByCodecName() if --video-encoder is specified.
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java226-254
Before the first frame is sent, if the encoder throws IllegalStateException or IllegalArgumentException, prepareRetry() attempts to reduce the capture size using a fixed fallback sequence: {2560, 1920, 1600, 1280, 1024, 800}. This calls capture.setMaxSize(newMaxSize) and retries. After the first frame has been sent, the encoder allows up to MAX_CONSECUTIVE_ERRORS = 3 errors before giving up.
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java149-195
CaptureReset (server/src/main/java/com/genymobile/scrcpy/video/CaptureReset.java) implements SurfaceCapture.CaptureListener and bridges a capture size change (e.g., display rotation) to a graceful encoder restart.
Flow:
SurfaceEncoder stores the current MediaCodec instance in reset.setRunningMediaCodec() so that CaptureReset can signal EOS immediately when invalidation occurs. The encode loop checks reset.consumeReset() before and after calling encode().
Sources: server/src/main/java/com/genymobile/scrcpy/video/CaptureReset.java1-37 server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java76-113
Before streaming video frames, the encoder transmits codec configuration data extracted from the first output buffer marked with BUFFER_FLAG_CODEC_CONFIG. The Streamer class sends this as a special packet to the client.
Configuration data by codec:
| Codec | Configuration Units |
|---|---|
| H.264 | SPS, PPS |
| H.265 | VPS, SPS, PPS |
| AV1 | Sequence Header |
In encode(), a buffer with BUFFER_FLAG_CODEC_CONFIG set is treated as a config packet (does not increment firstFrameSent). All other non-empty buffers are treated as video frames.
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java197-224
Streamer (in device/Streamer.java) receives ByteBuffer + MediaCodec.BufferInfo pairs from SurfaceEncoder.encode() via streamer.writePacket(). The wire format is:
[PTS: 8 bytes][Flags: 4 bytes][Length: 4 bytes][Payload: variable]
Packet flags:
FLAG_CODEC_CONFIG (0x1): Codec configuration data (SPS/PPS/VPS)FLAG_KEY_FRAME (0x2): Keyframe (I-frame)streamer.writeVideoHeader(size) sends the video dimensions before any packets, so the client's demuxer (sc_demuxer) knows the expected frame size.
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java197-224
Diagram: Full startup and streaming sequence
Sources: server/src/main/java/com/genymobile/scrcpy/video/SurfaceEncoder.java65-147 server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java104-161
options.getVideoBitRate() → MediaFormat.KEY_BIT_RATE--video-codec-options--max-fps maps to options.getMaxFps(), which sets the private max-fps-to-encoder key on the MediaFormatKEY_FRAME_RATE is hardcoded to 60 to satisfy the MediaCodec API requirement, but actual frame rate is variableKEY_REPEAT_PREVIOUS_FRAME_AFTER is set to 100 ms so the first frame appears immediately and quality recovers after idle periodsSize.limit(int maxSize) scales down the largest dimension to maxSize while preserving aspect ratio. Size.round8() then rounds both dimensions to a multiple of 8, as required by most hardware encoders.
Sources: server/src/main/java/com/genymobile/scrcpy/device/Size.java32-81
When a SurfaceCapture applies video transforms (crop, rotation, angle), the video displayed to the user is no longer a 1:1 pixel mapping of the device display. PositionMapper translates click coordinates from video space back into device display space.
PositionMapper.create() builds the full transform:
pixel_in_device = ndcToPixels(targetSize) × filterTransform⁻¹ × ndcFromPixels(videoSize)
map(Position position) returns null if position.getScreenSize() does not match the current videoSize, which rejects stale input events from a previous screen orientation.
| Field | Description |
|---|---|
videoSize | Current encoded video dimensions |
videoToDeviceMatrix | Composed affine transform from video pixel space to device pixel space |
Sources: server/src/main/java/com/genymobile/scrcpy/control/PositionMapper.java1-48
The server prioritizes hardware encoders for efficiency:
MediaCodecList for available encodersSpecific encoder can be forced via --video-encoder option with exact codec name.
Typical encoding latency breakdown:
Low-latency mode can reduce this via codec-specific options:
Video encoding is memory-intensive:
Sources: Architecture context from diagrams and general Android MediaCodec knowledge
Common encoder failure scenarios:
When client disconnects:
Sources: General error handling patterns from architecture context
Refresh this wiki
This wiki was recently refreshed. Please wait 3 days to refresh again.