VVC contains two-way optical flow (BDOF) tools. BDOF (formerly known as BIO) has been included in JEM. Compared with the JEM version, the BDOF in VVC is a simple version that requires fewer calculations in terms of the number of multiplications and the size of the multiplication.
1 Conditions of use
BDOF is used to optimize the bidirectional prediction signal of the CU at the 4x4 sub-block level. If BDOF meets all the following conditions, apply BDOF to CU:
1. Use "true" bi-prediction mode, that is, one of the two reference frames is before the current frame and the other is after the current frame;
2. Two The distance between two reference frames and the current frame (ie POC difference) is the same ;
3. Both reference frames are short-term reference frames;
4. CU does not use Affine mode or ATMVP mode for encoding;
5. CU requires more than 64 brightness pixel value (4x16 such undesirable);
6.CU height and width greater than or equal to 8 ;
7. for BCW weight: requirements and other weight ;
8. current CU is not weighted prediction is enabled;
9. current CU mode or not using CIIP TPM mode.
2 Calculation method
BDOF only applies to the luminance component . As the name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the motion of the object is smooth. For each 4x4 sub-block , the motion refinement is calculated by minimizing the difference between the L0 and L1 prediction samples (vx,vy), and then the motion refinement is used to adjust the bi-directional prediction sample value in the 4x4 sub-block. The following steps will be applied during the BDOF process:
2.1 Calculate horizontal and vertical gradient values
Calculation method: directly calculate the difference between two adjacent samples to calculate the gradient value of the two prediction signals, as follows:
where Iis the sample value kof the prediction signal coordinates (i,j)in the list , which is shift1calculated according to the luminance bit depth:
shift1 = max（6，bitDepth-6） //对于10比特深度一般为6
2.2 Calculating the autocorrelation and cross-correlation of gradient
The gradient autocorrelation and cross-correlation S1, S2, S3, S5, and S6 are calculated as follows:
Some of the parameters are defined as follows:
Among them Ωis a 6×6 window surrounding 4×4 sub-blocks, naand nbthe values of sum are respectively set as:
na = min（1，bitDepth-11）//对于10比特深度一般为-1
nb = min（4，bitDepth-8） //对于10比特深度一般为 2
2.3 Get refined motion
Using the above cross-correlation and auto-correlation S, derive the refined motion (vx,vy):
some of the parameters are defined as follows:
2.4 Get
Based on the motion refinement and gradient, the following adjustments are calculated for each sample in the 4×4 sub-block to obtain a low-order infinitesimal:
Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows:
these values can make the BDOF processing The multiplier of does not exceed 15 bits, and the maximum bit width of the intermediate parameter in BDOF processing is kept within 32 bits.
3 Implementation and expansion
In order to derive the gradient value, it is necessary to generate ksome predicted pixels in the list outside the boundary of the current CU I(the aforementioned 4x4 is extended to 6x6). As shown in the figure below,
BDOF in VVC uses an extended row/column around the CU boundary. In order to control the computational complexity of generating out-of-boundary prediction samples, reference pixels are directly obtained at nearby integer positions (using coordinates to calculate coordinates) to generate predicted pixels in the extended area (white position). There is no interpolation operation. The normal 8-tap motion compensation interpolation filter is used to generate prediction samples in the CU (gray position). These expanded pixel values are only used in gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values outside the CU boundary are needed, they are filled (ie repeated) from their nearest pixels.
When the width and/or height of a CU is greater than 16, it will be divided into sub-blocks with a width and/or height equal to 16 luminance samples, and the sub-block boundary is regarded as a CU boundary in the BDOF process. The maximum unit size of BDOF processing is limited to 16x16.
For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 and L1 prediction samples is less than the threshold, BDOF processing is not applied to the sub-block. The threshold is set equal to 8 x W x（H >> 1）, where W represents the sub-block width and H represents the sub-block height. In order to avoid the additional complexity of SAD calculation, the value calculated during the DVMR process of SAD between the initial L0 and L1 prediction samples is reused here.
If BCW is enabled for the current block, that is, the BCW weight index indicates that the weights are not equal, the bidirectional optical flow will be disabled. Similarly, if WP is enabled for the current block, that is, for any of the two reference pictures, the luma_weight_lx_flag flag is 1, then BDOF is also disabled. When the CU is encoded in symmetric MVD mode or CIIP mode, BDOF is also disabled.