How are mouse clicks actually communicated to the display? The diagram below breaks down the stages of the pipeline. Note that while there is overlap between the stages, they are guaranteed to start and complete in left-to-right order.
Let's introduce each of the boxes in the second row in the figure above individually. Note that box sizes are not to scale.
For simplicity, we'll focus on the mouse, but everything below applies to a wide variety of USB peripherals connected to the PC.
-
Mouse Hardware – This hardware is defined as the first electrical point of contact when the mouse is about to send events down the wire. In the mouse, there are several routines (such as anti-shake) that add a delay to the mouse button press event. Note that the anti-shake routine is important to prevent you from clicking the mouse when you don't want to. These extra clicks are often referred to as double clicks - two clicks are sent instead of one because the anti-shake routine is too aggressive. Latency isn't the only critical characteristic of mouse performance.
-
Mouse USB Hardware - Once debouncing is done, the mouse must wait for the next poll in order to send a packet down the wire. This time is reflected in the USB hardware.
-
Mouse USB Software – Mouse USB software refers to the time required by the operating system and mouse driver to process USB packets.
-
Sampling - Clicks go into the OS based on the mouse's polling rate, at which point they may wait for the next opportunity for the game to sample. This waiting time is called the sampling delay. This delay can be scaled up or down depending on the CPU frequency.
-
Simulation – The game must constantly update the state of the game world. This update is often referred to as emulation. Simulation includes work such as updating animations, game state, and changes caused by player input. Simulation is the stage where mouse input is applied to the game state.
-
Render Submit – As the simulation figures out where to place the relevant items in the next frame, it will start sending render jobs to the graphics API runtime. The runtime then passes rendering commands to the graphics driver.
-
Graphics Driver – The graphics driver is responsible for communicating with the GPU and sending it command packets. Depending on the graphics API, the driver may do this grouping for the developer, or the developer may be responsible for grouping rendering jobs.
-
Render Queue – After the driver submits a job for the GPU to execute, the job enters the render queue. The purpose of the render queue is to keep feeding the GPU with jobs by keeping the buffers filled with jobs that need to be executed by the GPU. This helps maximize FPS (throughput), but may introduce lag.
-
Rendering – The time it takes for the GPU to render all jobs related to a single frame.
-
Compositing – Depending on the display mode (fullscreen, borderless, windowed), the Desktop Window Manager (DWM) in the operating system must submit additional rendering jobs to composite the rest of the desktop for a particular frame. This may increase latency. It is recommended to always be in exclusive fullscreen mode to minimize compositing lag!
-
Scanout – Once compositing is complete, the final framebuffer is ready for display. The GPU will indicate that the framebuffer is ready to be displayed and replace the framebuffer to be read for scanout. If vsync is enabled, this "flip" in the framebuffer is stalled because it has to wait for the monitor's vsync signal. When ready, the GPU feeds the next frame to the display line by line according to the display's refresh rate (Hz). Since scanout is a function of refresh rate, we include it in Display Latency.
-
Display Processing – Display processing is the time it takes for a display to process an incoming frame (scanline) and initiate a pixel response.
-
Pixel Response – This refers to the time it takes for a pixel to change from one color to the next. Since the pixels are true liquid crystals, it takes time to change them. Pixel response time may depend on the intensity of the change required and will also depend on the panel technology.