For a long period of time, the refresh rate of mobile phones has been 60Hz. With the improvement of hardware device performance, various mobile devices with high refresh rates emerge in an endless stream, and mobile terminals can also have 120Hz display devices. So is the game on the mobile phone really the higher the FPS, the better? In this issue, we will explore the truth. Author: Hou Xin, Tencent game engine R & D engineers.
background
As mobile game developers, we spend a lot of time in our work trying to optimize our code. For example, let a certain piece of logic execute faster, or reduce the frequency of some iterations to reduce the burden on the CPU , or use various operations to reduce the burden on the GPU without reducing the picture quality .
The ultimate goal is to allow players with different performance devices to experience the game smoothly. "Stuck or not" is also one of the players' first impressions of the game. Therefore, our goal is to make the game run at the fastest speed.
Generally, the most intuitive indicator to evaluate whether a game is smooth is FPS (frames per second). So, is FPS really the higher the better?
The hardware performance evaluation on the market is to compare the FPS of the same quality game under different hardware, and the higher the better (Higher is better).
For the desktop platform, it has a continuous and stable power supply and a strong heat dissipation scheme, without considering the issues of heat and power consumption, allowing the hardware to play freely. Secondly, we have to consider the refresh rate (RefreshRate) of the display device (mobile screen, computer screen).
In order to understand the relationship between frame rate (in FPS) and refresh rate, let's take a look at their definitions:
1. Frame Rate (FrameRate)
The frame rate is the number of images that the GPU and CPU can produce when the game is running. The unit of measurement is FramePerSecond. It is usually an indicator for evaluating hardware performance and the smoothness of the game experience.
2. Refresh rate (RefreshRate)
The refresh rate (vertical refresh rate, vertical scan rate) indicates the number of new images that can be displayed by the display device in one second, and the unit of measurement is hertz (Hz).
Refresh rate and frame rate are two independent concepts. Frame rate represents the number of new images per second that the device driving the display can produce.
It can be simply understood as:
The game engine and driver are the producers, and the work efficiency is evaluated by the frame rate;
Display devices are consumers, and work efficiency is evaluated by refresh rate.
In short, the fluency we really feel will be limited by the refresh rate. When the frame rate is higher than the refresh rate, the number of images that the monitor can display per second remains the same.
3. ScreenTearing
Assuming that the refresh rate of your display device is 60Hz, when the frame rate is higher than the refresh rate or the frame time when the game is running is not a multiple of 1/60 (2/60, 3/60), that is, its FPS is not: .../120/60 /30/20/..., while the display is refreshing the image, new data is also being transferred from the graphics card, resulting in multiple frames of data appearing on the screen at the same time.
As shown in the above figure, frame B is rendered faster. When the data of frame A is still refreshed in the display, new data is submitted, causing screen tearing. This phenomenon is called screen tearing. The simplest solution to this defect is vertical synchronization (VSync).
4. Vertical Synchronization
Vertical synchronization will synchronize the work of the graphics card and the display device:
When the display is refreshing the data, the GPU will wait until the data is completely refreshed, and then the GPU will submit the new data and refresh it in the next refresh cycle.
Vertical synchronization will limit the FPS of the game to the refresh rate of the display device. The biggest problem is that it will cause player input delay because it requires the graphics card to wait for the display device to refresh the data after rendering.
Obviously, this problem has a great impact on competitive games.
（1）NVIDIA G-Sync
If you are an NVIDIA graphics card, and your graphics card and monitor both support G-Sync[1], the special chip in the monitor will communicate with the graphics card to make the monitor adjust its refresh rate to match the frame rate of the graphics card.
The disadvantage is that it requires hardware (display device hardware) support.
（2） AMD FreeSync
FreeSync [2] is based on the automatic synchronization technology [3] (Adaptive-Sync) supported by the DP interface (DisplayPort).
This is an open technical standard, so FreeSync does not require licensing fees, and display devices that support FreeSync are usually cheaper than G-Sync.
After understanding these factors that affect the user experience, we also understand the corresponding solutions. So, what is the specific situation of the mobile platform? How is it different from the desktop platform?
mobile platform
Taking Qualcomm as an example, the performance data of its Adreno[4] GPU is shown in the following table:
The Adreno 650 configured by Qualcomm Snapdragon 865 is compared with the Iris Plus attached to the Intel 11th generation CPU:
It can be seen that the floating point computing power of the current flagship mobile GPU has surpassed Intel's set display, the performance has been very impressive, and the mobile terminal hardware is also capable of outputting very high frame rates.
1. Android
The display devices of mobile platforms have been 60 Hz for a long time.
We have learned from the above that in the process of displaying game images on the screen, there is a synchronization process that affects the user experience.
There is a synchronization relationship between game logic and rendering loop and Android system and display hardware. This synchronization process is called Frame Pacing , which is the frame rate of the image generated by the engine, CPU, and GPU, and display refresh. Synchronization relationship between rates.
The Android display system can avoid the screen tearing (ScreenTearing) problem, that is, when the display is refreshing data, new data is pushed to the display device. It adopts the following measures to avoid tearing (Tearing):
Cache historical frame data;
Automatic detection of delayed frame data submission;
When there is a delay in submission, the historical frame data is rendered repeatedly.
Buffer the frame data through Buffer. When the display is refreshed, if there is new data transmission, just buffer it directly. With this design, there will be no blocking wait for VSYNC and no increase in input delay that affects game logic.
Although it brings a certain picture delay, it can avoid the problem of picture tearing. The specific data submission process is as follows:
First, the engine informs the display system SurfaceFlinger[5] through eglSwapBuffers, and then SurfaceFlinger buffers the data in the Buffer.
If the current refresh window period, SF will wait for the hardware VSYNC signal. After receiving the signal, SF will find the latest Buffer from the Buffer; if not found, use the last Buffer; if it is refreshing, SF is in a sleep state (device implementation related).
Assuming that the synchronization (Frame Pacing) frequency is 30Hz, the correct synchronization relationship is as follows:
NB means No Buffer, and Latch means Buffer transmission. The Display refresh rate in the figure is 60Hz, and the rendering frequency is 30Hz.
(1) Short frame freeze
When the rendering time of a frame becomes smaller, stuttering will occur:
As shown in the figure above, the rendering of the C frame takes a short time due to some reasons, and the rendering is completed in the next refresh window period. Therefore, the image data of the C frame is stored in the previous NB location, which eventually causes the refreshed frame sequence to change It is: AABCCC, the FrameTime of frame C is shorter, but the player feels that the game is not smooth.
(2) Solve short frame freeze
Android provides Swappy Frame Pacing library (part of Android Game SDK[6]), UE4.25[7] and Unity2019.2[8] have been merged into Swappy library.
Set the presentation time of the display through EGL_ANDROID_presentation_time[9]. In our example, the update frequency is 30Hz. By setting the PresentTime to 30Hz, the short frame freeze can be avoided.
(3) Long frame freeze/delay
As shown in the figure above, the B frame occupies more than 33.3ms of frame time due to some reasons, causing the NB frame to repeat twice, resulting in the AAABCC frame sequence, which leads to stutter:
A continues for three frames;
B only shows one frame.
(4) Solve the long frame freeze
Swappy will add a synchronization lock to make the display system have sufficient waiting space, so as not to affect the spread.
Through the synchronization lock EGL_KHR_fence_sync[10], although the problem of frame A cannot be solved, the B and C after frame A will not be affected by frame A.
2. Frame Pacing Library
The Swappy library in the Android Game SDK can not only solve the problem of long and short frames, but also support dynamically adjusting the refresh rate of the device to provide players with the smoothest visual experience. Devices with different refresh rates support different FPS:
60Hz：60FPS/30FPS/20FPS
60 + 90Hz：90FPS/60FPS/45FPS/30FPS
60 + 90 + 120Hz：120FPS/90FPS/60FPS/45FPS/40FPS/30FPS
Swappy can select the most suitable refresh rate according to the specific frame time of the renderer to provide players with a smoother visual experience. Through systrace[11], the improvement of the Frame Pacing library can be verified according to the data of SurfaceView.
So far we have learned that the Swappy library of Frame Pacing improvement scheme for the Android platform is actually a simplified version of G-Sync or Free-Sync. Both can dynamically adjust the display refresh rate (devices that support dynamic refresh rate) to output smoother effect
3. iOS
Apple shared a speech [12] at the 2018 WWDC, which introduced the improvements Apple has made on Frame Pacing.
In the above animation, although the left side is 40FPS, which is higher than the right side 30FPS, the user experience is obviously more friendly on the 30FPS side.
The execution timing of 40FPS is shown in the figure above, and the minimum interval of VSYNC is the refresh rate of 60Hz. When we render a new frame as fast as possible, there is no data in the cache of the 0/1 refresh point Display, so historical data is used.
That is, A shows 2 frames. In the second frame, the GPU calculation of frame B is completed, and B can be displayed directly. In the third frame, the GPU calculation of frame C is completed, and C is displayed directly, and because the GPU of A missed the refresh point 4, C also displayed two frames. Repeatedly in turn, caused: AABCCABBCAA long and short frame problems, which eventually led to the performance of lag.
(1) Set a fixed frame rate
iOS 10.3 and above support new API:
MTLDrawable addPresentedHandlerMTLCommandBuffer presentDrawable afterMinimumDurationMTLCommandBuffer presentDrawable atTime
The minimum frame interval can be set to solve the problem of long and short frames:
// Render Scene...// Get drawable and present at 30 FPSlet drawable = view.currentDrawable { // Render Final Pass ... let duration = 33.0 / 1000.0 // Duration of 33 ms commandBuffer.present(drawable, afterMinimumDuration: duration)}commandBuffer.commit()
By setting the minimum frame rendering interval, frames can be rendered at a fixed frequency for new frames, thereby leaving enough time for the CPU and GPU to render the scene.
Assuming that the refresh rate is 60Hz, as long as the time for the CPU and GPU to complete the collaborative output data is within 3*(1/60)ms, that is, the work C of the first frame GPU is guaranteed to be completed before the work A of the third frame is turned on, the iOS device You can output continuous 30Hz images.
4. Unreal Engine 4
Starting from version 4.25, UE4 integrates Android's Swappy library:
// Runtime/OpenGLDrv/Private/Android/AndroidOpenGLFramePacer.cppvoid FAndroidOpenGLFramePacer::Init() { InitSwappy();}void FAndroidOpenGLFramePacer::InitSwappy() { JNIEnv* Env = FAndroidApplication::GetJavaEnv(); SwappyGL_init(Env, FJavaWrapper::GameActivityThis);}
When Swappy is turned on, directly use its API to control Frame Pacing:
// r.setframepace 30// a.UseSwappyForFramePacing=1bool FAndroidOpenGLFramePacer::SwapBuffers(bool bLockToVsync) {#if USE_ANDROID_OPENGL_SWAPPY int64 DesiredFrameNS = (1000000000L) / (int64)FAndroidPlatformRHIFramePacer::GetFramePace(); SwappyGL_setSwapIntervalNS(DesiredFrameNS); SwappyGL_setAutoSwapInterval(false); SwappyGL_swap(eglDisplay, eglSurface);#endif
According to the code of UE4, UE4 does not use Swappy's default mode [13], but according to the configuration, the correct synchronization rhythm is set through Swappy.
Swappy understands Android better than UE4's default FramePacer. According to UE4's documentation, its real performance is also more stable than the default Pacer, and future versions will also use Swappy as the default FramePacer on the Android platform.
Versions below 4.25 use UE4's Leagcy Frame Pacer: the number of VBLANK[15] waits when controlling the Swap Buffer through eglSwapInterval[14].
VBLANK refers to the process from the last line of one frame of data to the beginning of the first line of data in the next frame. In fact, eglSwapInterval cannot accurately understand the refresh time of the display (hardware), so its real effect is not as good as Swappy, which knows more about the hardware. .
// rhi.SyncInterval, 60Hz设置为1int32 SyncInterval = GetLegacySyncInterval();// 当配置的同步间隔改变时，使用eglSwapInterval配置VBLANK等待次数if (DesiredSyncIntervalRelativeTo60Hz != SyncInterval) { eglSwapInterval(eglDisplay, DriverSyncIntervalRelativeToDevice);}
If the device supports the ANDROID_get_frame_timestamps[16] extension, some time data of the driver layer can be obtained through the API to calculate a more accurate SwapInterval:
EGLint Item = EGL_COMPOSITE_INTERVAL_ANDROID;// The time delta between subsequent composition events.eglGetCompositorTimingANDROID_p(eglDisplay, eglSurface, 1, &Item, &COMPOSITE_INTERVAL);if (COMPOSITE_INTERVAL >= 4000000 && COMPOSITE_INTERVAL <= 41666666) { DriverRefreshRate = float(1000000000.0 / double(COMPOSITE_INTERVAL)); DriverRefreshNanos = COMPOSITE_INTERVAL;}
And can solve the problem of short frame freeze. That is, by querying the data of the historical frame, the working timing of the Compositor is controlled. When a short frame occurs, the correct working time is calculated according to the refresh rate.
Ensure that the data B of the short frame is refreshed twice on the display to maintain the smoothness of the experience:
EGLint TimestampList = EGL_FIRST_COMPOSITION_START_TIME_ANDROID;// The first time at which// the compositor began preparing composition for this frame.EGLnsecsANDROID Result = 0;eglGetFrameTimestampsANDROID_p(eglDisplay, eglSurface, FrameIDs[Index % NUM_FRAMES_TO_MONITOR], 1, &TimestampList, &Result);// 设置下一帧的起始时间EGLnsecsANDROID DeltaNanos = EGLnsecsANDROID(DesiredSyncIntervalRelativeToDevice) * EGLnsecsANDROID(DeltaFrameIndex) * DriverRefreshNanos;EGLnsecsANDROID PresentationTime = Result + DeltaNanos;eglPresentationTimeANDROID_p(eglDisplay, eglSurface, PresentationTime);
（1）iOS
iOS controls the performance of FramePacing by controlling the parameters of CADisplayLink[17]:
FIOSPlatformRHIFramePacer::FrameInterval = NewFrameInterval;uint32 MaxRefreshRate = FIOSPlatformRHIFramePacer::GetMaxRefreshRate();CADisplayLink* displayLinkParam = (CADisplayLink*)param;// iOS 10displayLinkParam.preferredFramesPerSecond = MaxRefreshRate / FIOSPlatformRHIFramePacer::FrameInterval;// pre iOS 10displayLinkParam.frameInterval = FIOSPlatformRHIFramePacer::FrameInterval;
You can dynamically adjust FrameInterval or expected FPS based on historical frame data to achieve a smoother visual experience.
5. Unity
After Unity 2019.2, Swappy was integrated as FramePacer on the Android platform.
The Unity2018 version only sets glSwapInterval, which means that FramePacing is controlled through the inaccurate timestamp mode:
// Runtime/GfxDevice/egl/WindowContextEGL.cppEGLint WindowContextEGL::SetVSyncInterval(EGLint interval) { interval = clamp(interval, m_VSyncIntervalMin, m_VSyncIntervalMax); if (eglSwapInterval(m_EGLDisplay, interval)) return interval; return -1;}
The implementation of FramePacing on iOS is basically the same as UE4.
references:
[1] G-Sync:
https://www.nvidia.com/en-us/geforce/products/g-sync-monitors/
[2] FreeSync:
https://www.amd.com/en/technologies/free-sync
[3] Automatic synchronization technology:
https://vesa.org/featured-articles/vesa-adds-adaptive-sync-to-popular-displayport-video-standard/
[4] Adreno:
https://en.wikipedia.org/wiki/Adreno
[5] SurfaceFlinger:
https://source.android.com/devices/graphics/surfaceflinger-windowmanager
[6] Android Game SDK:
https://developer.android.com/games/sdk
[7] UE4.25:
https://docs.unrealengine.com/en-US/Platforms/Mobile/Rendering/MobileFramePacing/index.html
[8] Unity2019.2:
https://unity3d.com/unity/alpha/2019.2.0a6
[9] EGL_ANDROID_presentation_time:
https://www.khronos.org/registry/EGL/extensions/ANDROID/EGL_ANDROID_presentation_time.txt
[10] EGL_KHR_fence_sync:
https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_fence_sync.txt
[11] systrace:
https://developer.android.com/games/sdk/frame-pacing/opengl/verify-improvement
[12] WWDC speech:
https://developer.apple.com/videos/play/wwdc2018/612/
[13] Swappy default mode:
https://developer.android.com/games/sdk/frame-pacing#supported_operating_modes
[14] SwapInterval:
https://www.khronos.org/opengl/wiki/Swap_Interval
[15] VBLANK:
https://en.wikipedia.org/wiki/Vertical_blanking_interval
[16] ANDROID_get_frame_timestamps :
https://www.khronos.org/registry/EGL/extensions/ANDROID/EGL_ANDROID_get_frame_timestamps.txt
[17] CADisplayLink:
https://developer.apple.com/documentation/quartzcore/cadisplaylink