最近研究了VTM7.0中estIntraPredLumaQT函数的大体功能，之前大概看过一点，但是被这繁乱的代码搞自闭了，没有仔细研究过，这次借助H师兄的博客才勉强看懂大部分代码。
estIntraPredLumaQT函数主要是完成亮度分量的预测，并选出亮度预测的最佳模式。
主要流程如下：
一、初始化各种参数；
二、为了减少最终RDcost的次数，降低编码端的复杂度，estIntraPredLumaQT使用帧内快速搜索算法，主要包含三个阶段：
1、RMD：为了减少运算复杂度，直接对残差进行哈达玛变换求得SATD失真计算代价，将最小代价前几的模式加入全率失真模式列表中。
2、MPM：根据相邻块推导出当前块的最可能模式，加入到全率失真模式列表中。
3、RDO：遍历全率失真模式列表中的模式，进行完整的变换、量化、反变换、反量化等操作，计算失真和比特并求出RD Cost，选择代价最小的模式作为当前块的最优角度模式。
具体过程如下：
1、第一轮SATD粗选：对35种非扩展的传统的亮度帧内角度预测模式进行第一轮SATD循环粗选，这里粗选的时候使用predIntraAng函数计算预测值，然后计算SAD和SATD（HAD哈达玛变换后的SAD），并选其最小值计算每种模式的Cost，然后调用updateCandList函数更新全率失真候选模式列表uiRdModeList，候选列表长度为numModesForFullRD。
2、第二轮SATD粗选：
（1）对第一轮选出的numModesForFullRD种非扩展角度模式，分别对其左右相邻的扩展角度模式使用predIntraAng函数计算预测值，然后计算SAD和SATD，并选其最小值计算每种模式的Cost，然后调用updateCandList函数更新全率失真候选模式列表uiRdModeList，候选列表长度为numModesForFullRD。
（2）对传统的MPM列表进行SATD粗选，选出最有可能的几种模式加入到全率失真优化列表中。粗选过程与之前类似，循环遍历MPM列表中的每种模式，再调用predIntraAng函数计算预测值，然后计算SAD和SATD，并选其最小值计算每种模式的Cost，调用updateCandList函数更新全率失真候选模式列表uiRdModeList。
（3）使用哈达玛变换对MIP模式单独进行候选模式的推导。根据当前CU块的尺寸获取到MIP的模式数，然后对每一种模式调用predIntraMip函数计算预测值，然后计算SAD和SATD，并选其最小值计算每种模式的Cost，再调用updateCandList函数更新全率失真候选模式列表uiRdModeList。在这之后，调用reduceHadCandList函数对全率失真候选模式列表uiRdModeList进行缩减处理。
（4）获取FastUDI可能的MPM模式列表，遍历这些最可能的模式，判断它们是否在全率失真候选列表中，若不在则将其加入进去。同样地，对ISP模式列表m_ispCandListHor进行相同的处理。
（5）如果需要测试ISP模式，则需为ISP在全率失真候选模式列表中保留位置，即将所有的ISP模式添加进uiRdModeList。
3、第三轮RD Cost细选：遍历RD候选模式列表，如果当前模式为ISP模式，则将ISP列表重新排列（仅排列一次），然后调用xIntraCodingLumaISP函数进行ISP模式的残差变换、量化并计算RD Cost；否则调用xRecurIntraCodingLumaQT函数进行普通的残差变换、量化并计算RD Cost。
4、收尾工作：判断最优模式是否是MIP以及是否使用多参考行，并将最优模式保存入pu.intraDir里。
注意：这里涉及到许多快速算法
对于DCT-2变换和MTS变换，为降低复杂度，使用mtsUsageFlag变量标记MTS以实现快速算法。
- mtsUsageFlag=0时，表示MTS不适用于当前CU，即一次变换只需要检查DCT-2，此时需要进行第一轮SATD粗选和第二轮SATD粗选，并将粗选结果保存下来。
- mtsUsageFlag=1时，表示当前CU可以使用DCT-2和MTS，且当前正在检查DCT-2，此时需要进行第一轮SATD粗选和第二轮SATD粗选，并将粗选过程保存下来。
- mtsUsageFlag=2时，表示当前CU正在检查MTS，这里由于之前检查DCT-2的时候已经进行了一轮和二轮的SATD粗选，所以可以直接将粗选结果加载出来。
对于LFNST变换，当lfnstIdx=0时候需要进行完整的一轮、二轮粗选和RD Cost细选过程，并将第一轮SATD和第二轮SATD粗选的（1）（2）（3）过程的全率失真候选列表保存下来。当lfnstIdx=1、2时直接加载lfnst=0时保存的全率失真候选列表，并且不再将ISP模式添加进候选模式列表，这样可以大大降低编码端的复杂度（相关变量LFNSTSaveFlag、LFNSTLoadFlag）。
至于DCT-2变换、MTS变换和LFNST变换的遍历检查，在estIntraPredLumaQT的上层函数xCheckRDCostIntra中进行。
代码及注释如下：
bool IntraSearch::estIntraPredLumaQT( CodingUnit &cu, Partitioner &partitioner, const double bestCostSoFar, bool mtsCheckRangeFlag, int mtsFirstCheckId, int mtsLastCheckId, bool moreProbMTSIdxFirst )
{
CodingStructure &cs = *cu.cs;//获取当前CU的编码结构
const SPS &sps = *cs.sps;//获取该编码结构的语法元素
const uint32_t uiWidthBit = floorLog2(partitioner.currArea().lwidth() );
const uint32_t uiHeightBit = floorLog2(partitioner.currArea().lheight());
// Lambda calculation at equivalent Qp of 4 is recommended because at that Qp, the quantization divisor is 1.
//建议在等效qp为4时进行lambda的计算，因为在该qp时，用于量化的除数为1。
const double sqrtLambdaForFirstPass = m_pcRdCost->getMotionLambda(cu.transQuantBypass) * FRAC_BITS_SCALE;
//===== loop over partitions =====
//循环分区
//上下文模型的开始
const TempCtx ctxStart ( m_CtxCache, m_CABACEstimator->getCtx() );
//Mip模式的flag上下文模型
const TempCtx ctxStartMipFlag ( m_CtxCache, SubCtx( Ctx::MipFlag, m_CABACEstimator->getCtx() ) );
//ISP模式的上下文
const TempCtx ctxStartIspMode ( m_CtxCache, SubCtx( Ctx::ISPMode, m_CABACEstimator->getCtx() ) );
//Planar模式的flag上下文模型
const TempCtx ctxStartPlanarFlag ( m_CtxCache, SubCtx( Ctx::IntraLumaPlanarFlag, m_CABACEstimator->getCtx() ) );
//其他帧内模式的上下文
const TempCtx ctxStartIntraMode(m_CtxCache, SubCtx(Ctx::IntraLumaMpmFlag, m_CABACEstimator->getCtx()));
//多参考行的索引
const TempCtx ctxStartMrlIdx ( m_CtxCache, SubCtx( Ctx::MultiRefLineIdx, m_CABACEstimator->getCtx() ) );
//检查如果CU没有对应的PU块的则抛出异常
CHECK( !cu.firstPU, "CU has no PUs" );
const bool keepResi = cs.pps->getPpsRangeExtension().getCrossComponentPredictionEnabledFlag() || KEEP_PRED_AND_RESI_SIGNALS;
// variables for saving fast intra modes scan results across multiple LFNST passes
//用于保存多个LFNST遍的快速内部模式扫描结果的变量
//这两个变量主要用于使用LFNST时，只需要进行一遍一轮和二轮的SATD扫描
//即当lfnstIdx=0时，使用LFNSTSaveFlag将一轮和二轮扫描结果保存下来
//当lfnstIdx=1、2时，使用LFNSTLoadFLag直接加载一轮和二轮的扫描结果
bool LFNSTLoadFlag = sps.getUseLFNST() && cu.lfnstIdx != 0;//SPS层开启LFNST，且正在检查lfnst
bool LFNSTSaveFlag = sps.getUseLFNST() && cu.lfnstIdx == 0;//SPS层开启LFNST且没有检查LFNST
//若SPS层开启帧内MTS，判断当前是否检查的是DCT-2；若没有开启帧内MTS，则说明变换必是DCT2，故为true
LFNSTSaveFlag &= sps.getUseIntraMTS() ? cu.mtsFlag == 0 : true;
const uint32_t lfnstIdx = cu.lfnstIdx;
double costInterCU = findInterCUCost( cu );//帧间CU的cost
//当前亮度CU的宽和高
const int width = partitioner.currArea().lwidth();
const int height = partitioner.currArea().lheight();
// Marking MTS usage for faster MTS
// 0: MTS is either not applicable for current CU (cuWidth > MTS_INTRA_MAX_CU_SIZE or cuHeight > MTS_INTRA_MAX_CU_SIZE), not active in the config file or the fast decision algorithm is not used in this case
// 1: MTS fast algorithm can be applied for the current CU, and the DCT2 is being checked
// 2: MTS is being checked for current CU. Stored results of DCT2 can be utilized for speedup
// 标记MTS的使用以获得更快的MTS
// 0:MTS不适用于当前CU（cuWidth>MTS_INTRA_MAX_CU_SIZE或cuHeight>MTS_INTRA_MAX_CU_SIZE），在配置文件中不活动，或者在这种情况下不使用快速决策算法
// 1:当前CU可以采用MTS快速算法，正在检查DCT2
// 2:正在检查当前CU的MTS。DCT2的存储结果可用于加速
uint8_t mtsUsageFlag = 0;
const int maxSizeEMT = MTS_INTRA_MAX_CU_SIZE;
// SPS层开启帧内MTS，且宽和高均小于等于maxSizeEMT
if( width <= maxSizeEMT && height <= maxSizeEMT && sps.getUseIntraMTS() )
{
// SPS层开启LFNST，且当前检查的为MTS，则mtsUsageFlag为2；否则mtsUsageFlag为1
mtsUsageFlag = ( sps.getUseLFNST() && cu.mtsFlag == 1 ) ? 2 : 1;
}
//宽度*高度小于64 且没有开启LFNST快速算法
if( width * height < 64 && !m_pcEncCfg->getUseFastLFNST() )
{
mtsUsageFlag = 0;
}
//定义一个当前最优的代价
double bestCurrentCost = bestCostSoFar;
//当SPS层开启ISP且当前检查DCT-2，lfnstIdx = 0，并且满足ISP的尺寸使用条件，则可以测试ISP
bool testISP = sps.getUseISP() && cu.mtsFlag == 0 && cu.lfnstIdx == 0 && CU::canUseISP( width, height, cu.cs->sps->getMaxTbSize() );
if( testISP )
{
//reset the variables used for the tests
//重置用于测试的变量
m_ispCandListHor.clear();
m_ispCandListVer.clear();
m_regIntraRDListWithCosts.clear();
int numTotalPartsHor = (int)width >> floorLog2(CU::getISPSplitDim(width, height, TU_1D_VERT_SPLIT));//垂直划分所得子分区数
int numTotalPartsVer = (int)height >> floorLog2(CU::getISPSplitDim(width, height, TU_1D_HORZ_SPLIT));//水平划分所得子分区数
#if JVET_P1026_ISP_LFNST_COMBINATION
m_ispTestedModes[0].init( numTotalPartsHor, numTotalPartsVer );
//the total number of subpartitions is modified to take into account the cases where LFNST cannot be combined with ISP due to size restrictions
//修改子分区的总数，以考虑由于大小限制而无法将LFNST与ISP结合的情况
numTotalPartsHor = sps.getUseLFNST() && CU::canUseLfnstWithISP(cu.Y(), HOR_INTRA_SUBPARTITIONS) ? numTotalPartsHor : 0;
numTotalPartsVer = sps.getUseLFNST() && CU::canUseLfnstWithISP(cu.Y(), VER_INTRA_SUBPARTITIONS) ? numTotalPartsVer : 0;
for (int j = 1; j < NUM_LFNST_NUM_PER_SET; j++)
{
m_ispTestedModes[j].init(numTotalPartsHor, numTotalPartsVer);
}
#else
m_ispTestedModes.init(numTotalPartsHor, numTotalPartsVer);
#endif
}//if(testISP)
#if JVET_P0059_CHROMA_BDPCM
//BDPCM模式测试标志
const bool testBDPCM = (sps.getBDPCMEnabled()!=0) && CU::bdpcmAllowed(cu, ComponentID(partitioner.chType)) && cu.mtsFlag == 0 && cu.lfnstIdx == 0;
#else
const bool testBDPCM = sps.getBDPCMEnabledFlag() && CU::bdpcmAllowed( cu, ComponentID( partitioner.chType ) ) && cu.mtsFlag == 0 && cu.lfnstIdx == 0;
#endif
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> uiHadModeList;//哈达玛变换（即SATD）粗选后的候选列表
static_vector<double, FAST_UDI_MAX_RDMODE_NUM> CandCostList;//代价候选列表
static_vector<double, FAST_UDI_MAX_RDMODE_NUM> CandHadList;//哈达玛变换（SATD/HAD）后的候选代价列表
auto &pu = *cu.firstPU;
bool validReturn = false;
//代码块
{
CandHadList.clear();
CandCostList.clear();
uiHadModeList.clear();
CHECK(pu.cu != &cu, "PU is not contained in the CU");
//===== determine set of modes to be tested (using prediction signal only) =====
//===== 确定要测试的模式集（仅使用预测信号）======
int numModesAvailable = NUM_LUMA_MODE; // 所有67种传统帧内模式的数量，不包括MIP模式，MIP模式另行处理
const bool fastMip = sps.getUseMIP() && m_pcEncCfg->getUseFastMIP();//加快速编码版本的MIP模式
#if JVET_P0803_COMBINED_MIP_CLEANUP
const bool mipAllowed = sps.getUseMIP() && isLuma(partitioner.chType) && ((cu.lfnstIdx == 0) || allowLfnstWithMip(cu.firstPU->lumaSize()));//允许MIP模式使用标志
//是否是Mip模式的测试 //是否是Mip模式的测试 //是否是Mip模式的测试
const bool testMip = mipAllowed && !(cu.lwidth() > (8 * cu.lheight()) || cu.lheight() > (8 * cu.lwidth())) && !(cu.lwidth() > MIP_MAX_WIDTH || cu.lheight() > MIP_MAX_HEIGHT);
#else
const bool mipAllowed = sps.getUseMIP() && isLuma(partitioner.chType) && pu.lwidth() <= cu.cs->sps->getMaxTbSize() && pu.lheight() <= cu.cs->sps->getMaxTbSize() && ((cu.lfnstIdx == 0) || allowLfnstWithMip(cu.firstPU->lumaSize()));
const bool testMip = mipAllowed && mipModesAvailable(pu.Y());
#endif
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> uiRdModeList;//不使用哈达玛变换（即SAD）的RD候选列表
/****这里定义的就是能够进入到最终的RDcost环节的最优的几种模式的数量，numModesForFullRD初始化为3****/
int numModesForFullRD = 3;//最终的RDcost环节的最优的几种模式的数量
//根据CU的尺寸选取模式数量
numModesForFullRD = g_aucIntraModeNumFast_UseMPM_2D[uiWidthBit - MIN_CU_LOG2][uiHeightBit - MIN_CU_LOG2];
#if INTRA_FULL_SEARCH
numModesForFullRD = numModesAvailable;
#endif
//mtsUsageFlag为0或者为1，此时一次变换为DCT-2
if( mtsUsageFlag != 2 )
{
// this should always be true
CHECK( !pu.Y().valid(), "PU is not valid" );
//是否是CTU的第一行
bool isFirstLineOfCtu = (((pu.block(COMPONENT_Y).y)&((pu.cs->sps)->getMaxCUWidth() - 1)) == 0);
//使用的扩展参考行的数量，如果为CTU的第一行，则最多只能用1行；如果不是，则最多可以用三行
int numOfPassesExtendRef = (isFirstLineOfCtu ? 1 : MRL_NUM_REF_LINES);
pu.multiRefIdx = 0;
//numModesForFullRD的数量应该属少于所有可用的帧内模式的数量的，因此这里是主要的RD的入口
if( numModesForFullRD != numModesAvailable )
{
CHECK( numModesForFullRD >= numModesAvailable, "Too many modes for full RD search" );
const CompArea &area = pu.Y();//获取当前亮度PU的区域
PelBuf piOrg = cs.getOrgBuf(area);//获取当前亮度PU区域的亮度原始值
PelBuf piPred = cs.getPredBuf(area);//获取当前亮度PU区域的亮度预测值
DistParam distParamSad; // 定义一个使用SAD的失真参数
DistParam distParamHad;//定义一个使用SATD（加了哈达玛变换）的失真参数
if (cu.slice->getLmcsEnabledFlag() && m_pcReshape->getCTUFlag())
{
CompArea tmpArea(COMPONENT_Y, area.chromaFormat, Position(0, 0), area.size());
PelBuf tmpOrg = m_tmpStorageLCU.getBuf(tmpArea);
tmpOrg.copyFrom(piOrg);
tmpOrg.rspSignal(m_pcReshape->getFwdLUT());
//设置失真参数
m_pcRdCost->setDistParam(distParamSad, tmpOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y, false); // Use SAD cost
m_pcRdCost->setDistParam(distParamHad, tmpOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y, true); // Use HAD (SATD) cost
}
else
{
m_pcRdCost->setDistParam(distParamSad, piOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y, false); // Use SAD cost
m_pcRdCost->setDistParam(distParamHad, piOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y, true); // Use HAD (SATD) cost
}
distParamSad.applyWeight = false;
distParamHad.applyWeight = false;
if( testMip)
{
numModesForFullRD += fastMip? std::max(numModesForFullRD, floorLog2(std::min(pu.lwidth(), pu.lheight())) - 1) : numModesForFullRD;
}
//定义SATD变换候选的数量
const int numHadCand = (testMip ? 2 : 1) * 3;//若是MIP模式为6，否则为3
//*** Derive (regular) candidates using Hadamard
//*** 使用哈达吗变换的推导角度候选（即使用SATD）
cu.mipFlag = false;
//===== init pattern for luma prediction =====
//初始化亮度预测模式
initIntraPatternChType(cu, pu.Y(), true);
bool bSatdChecked[NUM_INTRA_MODE];//定义一个存放已经被STAD粗选过的帧内模式的bool数组
memset( bSatdChecked, 0, sizeof( bSatdChecked ) );//初始化为0
if( !LFNSTLoadFlag )
{
//首先对67种非扩展的传统的亮度帧内模式进行第一轮SATD循环粗选
for( int modeIdx = 0; modeIdx < numModesAvailable; modeIdx++ )
{
uint32_t uiMode = modeIdx;
Distortion minSadHad = 0;
// Skip checking extended Angular modes in the first round of SATD
// 在第一轮SATD中跳过检查扩展角模式
// 主要检查0、1、2、4、6....66这些模式
if( uiMode > DC_IDX && ( uiMode & 1 ) )
{
continue;
}
bSatdChecked[uiMode] = true;
//intraDir中存放的就是当前CU最终选中的预测模式
pu.intraDir[0] = modeIdx;
//初始化帧内预测参数
initPredIntraParams(pu, pu.Y(), sps);
if( useDPCMForFirstPassIntraEstimation( pu, uiMode ) )
{
encPredIntraDPCM( COMPONENT_Y, piOrg, piPred, uiMode );
}
else
{
//传统角度预测模式的函数入口，进行传统的角度预测
predIntraAng( COMPONENT_Y, piPred, pu);
}
// Use the min between SAD and HAD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
// 使用SAD和HAD之间的最小值作为代价标准(HAD其实就是SATD，是加了哈达玛变换后的SAD)
// SAD的缩放比例为2，与HAD的缩放比例一致。
minSadHad += std::min(distParamSad.distFunc(distParamSad)*2, distParamHad.distFunc(distParamHad));
// NB xFracModeBitsIntra will not affect the mode for chroma that may have already been pre-estimated.
//获取Mip模式flag的上下文模型
m_CABACEstimator->getCtx() = SubCtx( Ctx::MipFlag, ctxStartMipFlag );
//获取ISP模式的上下文模型
m_CABACEstimator->getCtx() = SubCtx( Ctx::ISPMode, ctxStartIspMode );
//获取亮度Planar模式的上下文模型
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
//获取亮度MPM列表的上下文模型
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
//获取多参考行的上下文模型
m_CABACEstimator->getCtx() = SubCtx( Ctx::MultiRefLineIdx, ctxStartMrlIdx );
uint64_t fracModeBits = xFracModeBitsIntra(pu, uiMode, CHANNEL_TYPE_LUMA);
//计算每种模式的Cost，比较相互之间的Cost进行模式更新
double cost = ( double ) minSadHad + (double)fracModeBits * sqrtLambdaForFirstPass;
DTRACE(g_trace_ctx, D_INTRA_COST, "IntraHAD: %u, %llu, %f (%d)\n", minSadHad, fracModeBits, cost, uiMode);
#if JVET_P0803_COMBINED_MIP_CLEANUP
//更新SAD之后的RD候选模式列表以及代价列表
updateCandList( ModeInfo( false, false, 0, NOT_INTRA_SUBPARTITIONS, uiMode ), cost, uiRdModeList, CandCostList, numModesForFullRD );
// 更新传统帧内预测模式的候选模式列表
// 就是更新SATD之后的uiHadModeList、CandHadList这些列表
updateCandList( ModeInfo( false, false, 0, NOT_INTRA_SUBPARTITIONS, uiMode ), double(minSadHad), uiHadModeList, CandHadList, numHadCand );
#else
updateCandList( ModeInfo(false, 0, NOT_INTRA_SUBPARTITIONS, uiMode), cost, uiRdModeList, CandCostList, numModesForFullRD );
updateCandList( ModeInfo(false, 0, NOT_INTRA_SUBPARTITIONS, uiMode), (double)minSadHad, uiHadModeList, CandHadList, numHadCand );
#endif
}//for( int modeIdx = 0; modeIdx < numModesAvailable; modeIdx++ )
if( !sps.getUseMIP() && LFNSTSaveFlag )
{
// save found best modes
//保存发现的最优的模式
m_uiSavedNumRdModesLFNST = numModesForFullRD;//第一轮SAD粗选后选中的相对最优模式的数量
m_uiSavedRdModeListLFNST = uiRdModeList;//被保存下来的RD模式的列表
m_dSavedModeCostLFNST = CandCostList;//被保存下来的候选代价的列表
// PBINTRA fast
m_uiSavedHadModeListLFNST = uiHadModeList;
m_dSavedHadListLFNST = CandHadList;
LFNSTSaveFlag = false;
}
} // LFNSTFlag if(!LFNSTLoadFlag)
if( !sps.getUseMIP() && LFNSTLoadFlag )
{
// restore saved modes
numModesForFullRD = m_uiSavedNumRdModesLFNST;
uiRdModeList = m_uiSavedRdModeListLFNST;
CandCostList = m_dSavedModeCostLFNST;
// PBINTRA fast
uiHadModeList = m_uiSavedHadModeListLFNST;
CandHadList = m_dSavedHadListLFNST;
} // !LFNSTFlag
if (!(sps.getUseMIP() && LFNSTLoadFlag))
{
// 将第一轮选出来的最优模式放到父模式中
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> parentCandList = uiRdModeList;
// Second round of SATD for extended Angular modes
// 这里开始第二轮对剩下的扩展角度模式的SATD粗选
for (int modeIdx = 0; modeIdx < numModesForFullRD; modeIdx++)
{
// 定义父模式，父模式放入的是第一轮选出来的最优的模式列表
unsigned parentMode = parentCandList[modeIdx].modeId;
// 从扩展模式开始循环处理
// 模式号是从3-65
if (parentMode > (DC_IDX + 1) && parentMode < (NUM_LUMA_MODE - 1))
{
for (int subModeIdx = -1; subModeIdx <= 1; subModeIdx += 2)
{
unsigned mode = parentMode + subModeIdx;//给父模式减一或者加一，正好就是上一轮那些最优候选列表中非扩展角度模式的左右相邻的扩展角度模式
if (!bSatdChecked[mode])//如果mode没有进行过SATD粗选
{
pu.intraDir[0] = mode;
initPredIntraParams(pu, pu.Y(), sps);
if (useDPCMForFirstPassIntraEstimation(pu, mode))
{
encPredIntraDPCM(COMPONENT_Y, piOrg, piPred, mode);
}
else
{
predIntraAng(COMPONENT_Y, piPred, pu );
}
// Use the min between SAD and SATD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
// 使用SAD和SATD之间的最小值作为代价标准
// SAD按2缩放以与HAD的缩放对齐
Distortion minSadHad = std::min(distParamSad.distFunc(distParamSad)*2, distParamHad.distFunc(distParamHad));
// NB xFracModeBitsIntra will not affect the mode for chroma that may have already been pre-estimated.
m_CABACEstimator->getCtx() = SubCtx( Ctx::MipFlag, ctxStartMipFlag );
m_CABACEstimator->getCtx() = SubCtx( Ctx::ISPMode, ctxStartIspMode );
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
m_CABACEstimator->getCtx() = SubCtx( Ctx::MultiRefLineIdx, ctxStartMrlIdx );
uint64_t fracModeBits = xFracModeBitsIntra(pu, mode, CHANNEL_TYPE_LUMA);
// 计算第二轮SATD每一种模式的cost
double cost = (double) minSadHad + (double) fracModeBits * sqrtLambdaForFirstPass;
#if JVET_P0803_COMBINED_MIP_CLEANUP
// 更新SAD之后的RD候选模式列表以及代价列表
updateCandList( ModeInfo( false, false, 0, NOT_INTRA_SUBPARTITIONS, mode ), cost, uiRdModeList, CandCostList, numModesForFullRD );
// 更新模式候选列表，将最优的扩展角度模式加到模式列表中
updateCandList( ModeInfo( false, false, 0, NOT_INTRA_SUBPARTITIONS, mode ), double(minSadHad), uiHadModeList, CandHadList, numHadCand );
#else
updateCandList( ModeInfo( false, 0, NOT_INTRA_SUBPARTITIONS, mode ), cost, uiRdModeList, CandCostList, numModesForFullRD );
updateCandList( ModeInfo( false, 0, NOT_INTRA_SUBPARTITIONS, mode ), (double)minSadHad, uiHadModeList, CandHadList, numHadCand );
#endif
bSatdChecked[mode] = true;
}
}
}
}//for (int modeIdx = 0; modeIdx < numModesForFullRD; modeIdx++)
if ( testISP )
{
// we save the regular intra modes list
// 我们保存常规帧内模式列表
m_ispCandListHor = uiRdModeList;
}
pu.multiRefIdx = 1;//使用1参考行
const int numMPMs = NUM_MOST_PROBABLE_MODES;//定义最可能模式的数量，即MPM列表的大小
unsigned multiRefMPM [numMPMs];//定义多参考行的MPM列表
PU::getIntraMPMs(pu, multiRefMPM);//获取该MPM列表的函数入口
// 对每个扩展的参考行进行循环
for (int mRefNum = 1; mRefNum < numOfPassesExtendRef; mRefNum++)
{
// 定义多参考行的索引，不同参考行的MPM列表是不同的
int multiRefIdx = MULTI_REF_LINE_IDX[mRefNum];
pu.multiRefIdx = multiRefIdx;
{
initIntraPatternChType(cu, pu.Y(), true);
}
// 对MPM列表中的6个模式进行SATD比较选择
for (int x = 1; x < numMPMs; x++)
{
uint32_t mode = multiRefMPM[x];
{
pu.intraDir[0] = mode;
initPredIntraParams(pu, pu.Y(), sps);
if (useDPCMForFirstPassIntraEstimation(pu, mode))
{
encPredIntraDPCM(COMPONENT_Y, piOrg, piPred, mode);
}
else
{
predIntraAng(COMPONENT_Y, piPred, pu);
}
// Use the min between SAD and SATD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
Distortion minSadHad = std::min(distParamSad.distFunc(distParamSad)*2, distParamHad.distFunc(distParamHad));
// NB xFracModeBitsIntra will not affect the mode for chroma that may have already been pre-estimated.
m_CABACEstimator->getCtx() = SubCtx( Ctx::MipFlag, ctxStartMipFlag );
m_CABACEstimator->getCtx() = SubCtx( Ctx::ISPMode, ctxStartIspMode );
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
m_CABACEstimator->getCtx() = SubCtx( Ctx::MultiRefLineIdx, ctxStartMrlIdx );
uint64_t fracModeBits = xFracModeBitsIntra(pu, mode, CHANNEL_TYPE_LUMA);
double cost = (double)minSadHad + (double)fracModeBits * sqrtLambdaForFirstPass;
#if JVET_P0803_COMBINED_MIP_CLEANUP
//更新候选列表
updateCandList( ModeInfo( false, false, multiRefIdx, NOT_INTRA_SUBPARTITIONS, mode ), cost, uiRdModeList, CandCostList, numModesForFullRD );
updateCandList( ModeInfo( false, false, multiRefIdx, NOT_INTRA_SUBPARTITIONS, mode ), double(minSadHad), uiHadModeList, CandHadList, numHadCand );
#else
updateCandList( ModeInfo( false, multiRefIdx, NOT_INTRA_SUBPARTITIONS, mode ), cost, uiRdModeList, CandCostList, numModesForFullRD );
updateCandList( ModeInfo( false, multiRefIdx, NOT_INTRA_SUBPARTITIONS, mode ), (double)minSadHad, uiHadModeList, CandHadList, numHadCand );
#endif
}
}//for (int x = 1; x < numMPMs; x++)
}//for (int mRefNum = 1; mRefNum < numOfPassesExtendRef; mRefNum++)
CHECKD( uiRdModeList.size() != numModesForFullRD, "Error: RD mode list size" );
//为下次运行保存不同的集
if (LFNSTSaveFlag && testMip && !allowLfnstWithMip(cu.firstPU->lumaSize())) // save a different set for the next run
{
// save found best modes
m_uiSavedRdModeListLFNST = uiRdModeList;
m_dSavedModeCostLFNST = CandCostList;
// PBINTRA fast
m_uiSavedHadModeListLFNST = uiHadModeList;
m_dSavedHadListLFNST = CandHadList;
m_uiSavedNumRdModesLFNST = g_aucIntraModeNumFast_UseMPM_2D[uiWidthBit - MIN_CU_LOG2][uiHeightBit - MIN_CU_LOG2];
m_uiSavedRdModeListLFNST.resize(m_uiSavedNumRdModesLFNST);
m_dSavedModeCostLFNST.resize(m_uiSavedNumRdModesLFNST);
// PBINTRA fast
m_uiSavedHadModeListLFNST.resize(3);
m_dSavedHadListLFNST.resize(3);
LFNSTSaveFlag = false;
}
//*** Derive MIP candidates using Hadamard
//在两轮传统角度模式的SATD以及MPM的SATD结束以后，对使用哈达玛的MIP模式单独进行候选模式的推导
if (testMip)
{
cu.mipFlag = true;
pu.multiRefIdx = 0;
double mipHadCost[MAX_NUM_MIP_MODE] = { MAX_DOUBLE };
initIntraPatternChType(cu, pu.Y());
#if JVET_P0803_COMBINED_MIP_CLEANUP
// 初始化Mip模式
initIntraMip( pu, pu.Y() );
const int transpOff = getNumModesMip( pu.Y() );
const int numModesFull = (transpOff << 1);
// getNumModesMip函数根据当前CU的块尺寸获取不同的Mip模式数量
for( uint32_t uiModeFull = 0; uiModeFull < numModesFull; uiModeFull++ )
{
const bool isTransposed = (uiModeFull >= transpOff ? true : false);
const uint32_t uiMode = (isTransposed ? uiModeFull - transpOff : uiModeFull);
pu.mipTransposedFlag = isTransposed;
#else
initIntraMip( pu );
for (uint32_t uiMode = 0; uiMode < getNumModesMip(pu.Y()); uiMode++)
{
#endif
pu.intraDir[CHANNEL_TYPE_LUMA] = uiMode;
// Mip预测模式的函数入口，进行MIP预测
predIntraMip(COMPONENT_Y, piPred, pu);
// Use the min between SAD and HAD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
Distortion minSadHad = std::min(distParamSad.distFunc(distParamSad)*2, distParamHad.distFunc(distParamHad));
m_CABACEstimator->getCtx() = SubCtx( Ctx::MipFlag, ctxStartMipFlag );
uint64_t fracModeBits = xFracModeBitsIntra(pu, uiMode, CHANNEL_TYPE_LUMA);
double cost = double(minSadHad) + double(fracModeBits) * sqrtLambdaForFirstPass;
#if JVET_P0803_COMBINED_MIP_CLEANUP
mipHadCost[uiModeFull] = cost;
DTRACE(g_trace_ctx, D_INTRA_COST, "IntraMIP: %u, %llu, %f (%d)\n", minSadHad, fracModeBits, cost, uiModeFull);
// 更新候选列表，将最优的Mip模式加入到RDCost的候选列表中
updateCandList( ModeInfo( true, isTransposed, 0, NOT_INTRA_SUBPARTITIONS, uiMode ), cost, uiRdModeList, CandCostList, numModesForFullRD + 1 );
updateCandList( ModeInfo( true, isTransposed, 0, NOT_INTRA_SUBPARTITIONS, uiMode ), 0.8*double(minSadHad), uiHadModeList, CandHadList, numHadCand );
#else
mipHadCost[uiMode] = cost;
DTRACE(g_trace_ctx, D_INTRA_COST, "IntraMIP: %u, %llu, %f (%d)\n", minSadHad, fracModeBits, cost, uiMode);
updateCandList(ModeInfo(true, 0, NOT_INTRA_SUBPARTITIONS, uiMode), cost, uiRdModeList, CandCostList, numModesForFullRD + 1);
updateCandList(ModeInfo(true, 0, NOT_INTRA_SUBPARTITIONS, uiMode), 0.8*double(minSadHad), uiHadModeList, CandHadList, numHadCand);
#endif
}
// 在总共四轮的STAD之后，最终更新好RDcost列表后，还要对该列表进行一个缩减的处理
const double thresholdHadCost = 1.0 + 1.4 / sqrt((double)(pu.lwidth()*pu.lheight()));
reduceHadCandList(uiRdModeList, CandCostList, numModesForFullRD, thresholdHadCost, mipHadCost, pu, fastMip);
}
if ( sps.getUseMIP() && LFNSTSaveFlag)
{
// save found best modes
m_uiSavedNumRdModesLFNST = numModesForFullRD;
m_uiSavedRdModeListLFNST = uiRdModeList;
m_dSavedModeCostLFNST = CandCostList;
// PBINTRA fast
m_uiSavedHadModeListLFNST = uiHadModeList;
m_dSavedHadListLFNST = CandHadList;
LFNSTSaveFlag = false;
}
}//if (!(sps.getUseMIP() && LFNSTLoadFlag))
else //if( sps.getUseMIP() && LFNSTLoadFlag)
{
// restore saved modes加载保存的模式
numModesForFullRD = m_uiSavedNumRdModesLFNST;
uiRdModeList = m_uiSavedRdModeListLFNST;
CandCostList = m_dSavedModeCostLFNST;
// PBINTRA fast
uiHadModeList = m_uiSavedHadModeListLFNST;
CandHadList = m_dSavedHadListLFNST;
}
//获取FastUDI可能的MPM列表，如果满足条件也将其加入RD候选列表
if( m_pcEncCfg->getFastUDIUseMPMEnabled() )
{
const int numMPMs = NUM_MOST_PROBABLE_MODES;
unsigned uiPreds[numMPMs];
pu.multiRefIdx = 0;
const int numCand = PU::getIntraMPMs( pu, uiPreds );//定义MPM列表中最可能的模式数量,然后只对MPM中最可能的那几个模式进行SATD粗选，其余模式不进行
for( int j = 0; j < numCand; j++ )
{
bool mostProbableModeIncluded = false;
#if JVET_P0803_COMBINED_MIP_CLEANUP
ModeInfo mostProbableMode( false, false, 0, NOT_INTRA_SUBPARTITIONS, uiPreds[j] );
#else
ModeInfo mostProbableMode( false, 0, NOT_INTRA_SUBPARTITIONS, uiPreds[j] );
#endif
for( int i = 0; i < numModesForFullRD; i++ )
{
mostProbableModeIncluded |= ( mostProbableMode == uiRdModeList[i] );
}
if( !mostProbableModeIncluded )
{
numModesForFullRD++;
uiRdModeList.push_back( mostProbableMode );
CandCostList.push_back(0);
}
}
if ( testISP )
{
// we add the MPMs to the list that contains only regular intra modes
// 将不加入多参考情况的ISP模式添加到RD候选列表中
for (int j = 0; j < numCand; j++)
{
bool mostProbableModeIncluded = false;
#if JVET_P0803_COMBINED_MIP_CLEANUP
ModeInfo mostProbableMode( false, false, 0, NOT_INTRA_SUBPARTITIONS, uiPreds[j] );
#else
ModeInfo mostProbableMode(false, 0, NOT_INTRA_SUBPARTITIONS, uiPreds[j]);
#endif
for (int i = 0; i < m_ispCandListHor.size(); i++)
{
mostProbableModeIncluded |= (mostProbableMode == m_ispCandListHor[i]);
}
if (!mostProbableModeIncluded)
{
m_ispCandListHor.push_back(mostProbableMode);
}
}
}
}//if( m_pcEncCfg->getFastUDIUseMPMEnabled() )
}//if( numModesForFullRD != numModesAvailable )
else
{
THROW( "Full search not supported for MIP" );
}
if( sps.getUseLFNST() && mtsUsageFlag == 1 )
{
// Store the modes to be checked with RD
m_savedNumRdModes[ lfnstIdx ] = numModesForFullRD;
std::copy_n( uiRdModeList.begin(), numModesForFullRD, m_savedRdModeList[ lfnstIdx ] );
}
}//if( mtsUsageFlag!=2 )
else //mtsUsage = 2 (here we potentially reduce the number of modes that will be full-RD checked)
{ //变换采用MTS，在这里，我们可能会减少将被完全RD检查的模式的数量
//如果使用帧内快速LFNST
if( ( m_pcEncCfg->getUseFastLFNST() || !cu.slice->isIntra() ) && m_bestModeCostValid[ lfnstIdx ] )
{
//重置RD候选模式列表的数量，下面进行第二次RD候选列表的缩减
numModesForFullRD = 0;
//设置一个跳过模式阈值
double thresholdSkipMode = 1.0 + ( ( cu.lfnstIdx > 0 ) ? 0.1 : 1.0 ) * ( 1.4 / sqrt( ( double ) ( width*height ) ) );
// Skip checking the modes with much larger R-D cost than the best mode
// 对所有RD候选模式进行循环处理，但是跳过检查比最佳模式代价大得多的R-D模式，不进行RDcost检查
for( int i = 0; i < m_savedNumRdModes[ lfnstIdx ]; i++ )
{
//如果该模式的代价小于阈值，则将其保留在RD候选模式列表中，否则将其排除
if( m_modeCostStore[ lfnstIdx ][ i ] <= thresholdSkipMode * m_bestModeCostStore[ lfnstIdx ] )
{
uiRdModeList.push_back( m_savedRdModeList[ lfnstIdx ][ i ] );
numModesForFullRD++;//RD候选模式列表的数量加一
}
}
}
else //this is necessary because we skip the candidates list calculation, since it was already obtained for the DCT-II. Now we load it
{//这里的操作是必要的，因为已经从DCT-2中获得了计算数据，因此这里跳过了候选列表的计算，现在我们加载它
// Restore the modes to be checked with RD
// 重新保存用于RD的候选模式数量
numModesForFullRD = m_savedNumRdModes[ lfnstIdx ];
// 重置RD候选列表的大小
uiRdModeList.resize( numModesForFullRD );
std::copy_n( m_savedRdModeList[ lfnstIdx ], m_savedNumRdModes[ lfnstIdx ], uiRdModeList.begin() );
CandCostList.resize( numModesForFullRD );
}
}//if(mtsUsage = 2)
CHECK( numModesForFullRD != uiRdModeList.size(), "Inconsistent state!" );
// after this point, don't use numModesForFullRD
// PBINTRA fast
// PBINTRA fast快速帧内模式
if( m_pcEncCfg->getUsePbIntraFast() && !cs.slice->isIntra() && uiRdModeList.size() < numModesAvailable && !cs.slice->getDisableSATDForRD() && ( mtsUsageFlag != 2 || lfnstIdx > 0 ) )
{
double pbintraRatio = (lfnstIdx > 0) ? 1.25 : PBINTRA_RATIO;
int maxSize = -1;
ModeInfo bestMipMode;
int bestMipIdx = -1;
for( int idx = 0; idx < uiRdModeList.size(); idx++ )
{
if( uiRdModeList[idx].mipFlg )
{
bestMipMode = uiRdModeList[idx];
bestMipIdx = idx;
break;
}
}
const int numHadCand = 3;
for (int k = numHadCand - 1; k >= 0; k--)
{
if (CandHadList.size() < (k + 1) || CandHadList[k] > cs.interHad * pbintraRatio) { maxSize = k; }
}
if (maxSize > 0)
{
uiRdModeList.resize(std::min<size_t>(uiRdModeList.size(), maxSize));
if( bestMipIdx >= 0 )
{
if( uiRdModeList.size() <= bestMipIdx )
{
uiRdModeList.push_back(bestMipMode);
}
}
if ( testISP )
{
m_ispCandListHor.resize(std::min<size_t>(m_ispCandListHor.size(), maxSize));
}
}
if (maxSize == 0)
{
cs.dist = std::numeric_limits<Distortion>::max();
cs.interHad = 0;
//===== reset context models =====
m_CABACEstimator->getCtx() = SubCtx(Ctx::MipFlag, ctxStartMipFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::ISPMode, ctxStartIspMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::MultiRefLineIdx, ctxStartMrlIdx);
return false;
}
}//PBINTRA fast
int numNonISPModes = (int)uiRdModeList.size();
if ( testISP ) //测试ISP
{
// we reserve positions for ISP in the common full RD list
// 我们为ISP在通用的完整RD列表中保留位置
#if JVET_P1026_ISP_LFNST_COMBINATION
const int maxNumRDModesISP = sps.getUseLFNST() ? 16 * NUM_LFNST_NUM_PER_SET : 16;
m_curIspLfnstIdx = 0;
#else
const int maxNumRDModesISP = 16;
#endif
for (int i = 0; i < maxNumRDModesISP; i++)
#if JVET_P0803_COMBINED_MIP_CLEANUP
uiRdModeList.push_back( ModeInfo( false, false, 0, INTRA_SUBPARTITIONS_RESERVED, 0 ) );
#else
uiRdModeList.push_back(ModeInfo(false, 0, INTRA_SUBPARTITIONS_RESERVED, 0));
#endif
}
//===== check modes (using r-d costs) =====
//check 所有模式（使用RDcost）
ModeInfo uiBestPUMode;//定义当前PU最优的亮度预测帧内模式
int bestBDPCMMode = 0;
double bestCostNonBDPCM = MAX_DOUBLE;//定义再不使用BDPCM模式的时候的最优代价
//临时缓存CS
CodingStructure *csTemp = m_pTempCS[gp_sizeIdxInfo->idxFrom( cu.lwidth() )][gp_sizeIdxInfo->idxFrom( cu.lheight() )];
//最好的模式缓存CS
CodingStructure *csBest = m_pBestCS[gp_sizeIdxInfo->idxFrom( cu.lwidth() )][gp_sizeIdxInfo->idxFrom( cu.lheight() )];
csTemp->slice = cs.slice;
csBest->slice = cs.slice;
csTemp->initStructData();
csBest->initStructData();
csTemp->picture = cs.picture;
csBest->picture = cs.picture;
#if !JVET_P0803_COMBINED_MIP_CLEANUP
static_vector<int, FAST_UDI_MAX_RDMODE_NUM> rdModeIdxList;
if (testMip)
{
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> rdModeListTemp;
for( int i = 0; i < uiRdModeList.size(); i++)
{
if( !uiRdModeList[i].mipFlg && uiRdModeList[i].ispMod==NOT_INTRA_SUBPARTITIONS )
{
rdModeListTemp.push_back( uiRdModeList[i] );
rdModeIdxList.push_back( i );
}
}
for( int i = 0; i < uiRdModeList.size(); i++)
{
if( uiRdModeList[i].mipFlg || uiRdModeList[i].ispMod!=NOT_INTRA_SUBPARTITIONS )
{
rdModeListTemp.push_back( uiRdModeList[i] );
rdModeIdxList.push_back( i );
}
}
uiRdModeList.resize(rdModeListTemp.size());
for( int i = 0; i < uiRdModeList.size(); i++)
{
uiRdModeList[i] = rdModeListTemp[i];
}
}
else
{
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> rdModeListTemp;
for( int i = 0; i < uiRdModeList.size(); i++ )
{
if( !uiRdModeList[i].mipFlg )
{
rdModeListTemp.push_back( uiRdModeList[i] );
}
}
uiRdModeList.resize(rdModeListTemp.size());
for( int i = 0; i < rdModeListTemp.size(); i++ )
{
uiRdModeList[i] = rdModeListTemp[i];
}
}
#endif
// just to be sure
numModesForFullRD = ( int ) uiRdModeList.size();
TUIntraSubPartitioner subTuPartitioner( partitioner );
#if JVET_P1026_ISP_LFNST_COMBINATION
if ( testISP )
{
m_modeCtrl->setIspCost( MAX_DOUBLE );
m_modeCtrl->setMtsFirstPassNoIspCost( MAX_DOUBLE );
}
int bestLfnstIdx = cu.lfnstIdx;
#else
if( !cu.ispMode && !cu.mtsFlag )
{
m_modeCtrl->setMtsFirstPassNoIspCost( MAX_DOUBLE );
}
#endif
/********************这里才进入到最终对于RD候选列表中的模式的进一步精选RDcost操作***********/
for (int mode = -2 * int(testBDPCM); mode < (int)uiRdModeList.size(); mode++)
{
// set CU/PU to luma prediction mode
// 首先设置亮度CU/PU的预测模式
ModeInfo uiOrgMode;
if ( mode < 0 )//如果是BDPCM模式
{
cu.bdpcmMode = -mode;//将负的模式号转化为正的
#if JVET_P0803_COMBINED_MIP_CLEANUP
uiOrgMode = ModeInfo( false, false, 0, NOT_INTRA_SUBPARTITIONS, cu.bdpcmMode == 2 ? VER_IDX : HOR_IDX );
#else
uiOrgMode = ModeInfo(false, 0, NOT_INTRA_SUBPARTITIONS, cu.bdpcmMode == 2 ? VER_IDX : HOR_IDX);
#endif
cu.mipFlag = uiOrgMode.mipFlg;
#if JVET_P0803_COMBINED_MIP_CLEANUP
pu.mipTransposedFlag = uiOrgMode.mipTrFlg;
#endif
cu.ispMode = uiOrgMode.ispMod;
pu.multiRefIdx = uiOrgMode.mRefId;
pu.intraDir[CHANNEL_TYPE_LUMA] = uiOrgMode.modeId;
}//if BDPCM
else
{//如果不是BDPCM模式
cu.bdpcmMode = 0;
if (uiRdModeList[mode].ispMod == INTRA_SUBPARTITIONS_RESERVED)//如果是ISP模式
{
if (mode == numNonISPModes) // the list needs to be sorted only once 列表仅重排一次
{
#if JVET_P1026_ISP_LFNST_COMBINATION
if (m_pcEncCfg->getUseFastISP())
{
m_modeCtrl->setBestPredModeDCT2( uiBestPUMode.modeId );
}
if (!xSortISPCandList(bestCurrentCost, csBest->cost, uiBestPUMode))//重新排列ISP列表，水平和垂直列表中的模式一样
break;
#else
xSortISPCandList(bestCurrentCost, csBest->cost);
#endif
}
//获取下一个ISP模式
xGetNextISPMode(uiRdModeList[mode], (mode > 0 ? &uiRdModeList[mode - 1] : nullptr), Size(width, height));
if (uiRdModeList[mode].ispMod == INTRA_SUBPARTITIONS_RESERVED)
continue;
#if JVET_P1026_ISP_LFNST_COMBINATION
cu.lfnstIdx = m_curIspLfnstIdx;
#endif
}//if ispMode
uiOrgMode = uiRdModeList[mode];//取出RD候选列表中的模式
cu.mipFlag = uiOrgMode.mipFlg;//当前候选模式是否是Mip模式
#if JVET_P0803_COMBINED_MIP_CLEANUP
pu.mipTransposedFlag = uiOrgMode.mipTrFlg;
#endif
cu.ispMode = uiOrgMode.ispMod;//当前候选模式是否是ISP模式
pu.multiRefIdx = uiOrgMode.mRefId;//当前候选模式是否是使用MRL
pu.intraDir[CHANNEL_TYPE_LUMA] = uiOrgMode.modeId;//当前候选模式放入intraDir
//MIP and MRL的结合不被允许
CHECK(cu.mipFlag && pu.multiRefIdx, "Error: combination of MIP and MRL not supported");
//Planar and MRL的结合不被允许
CHECK(pu.multiRefIdx && (pu.intraDir[0] == PLANAR_IDX), "Error: combination of MRL and Planar mode not supported");
//ISP and MIP的结合不被允许
CHECK(cu.ispMode && cu.mipFlag, "Error: combination of ISP and MIP not supported");
//ISP and MRL的结合不被允许
CHECK(cu.ispMode && pu.multiRefIdx, "Error: combination of ISP and MRL not supported");
}//if !BDPCM
// set context models
// 设置上下文模型
m_CABACEstimator->getCtx() = ctxStart;
// determine residual for partition
// 确定分区的残差
cs.initSubStructure( *csTemp, partitioner.chType, cs.area, true );
bool tmpValidReturn = false;
if( cu.ispMode )//如果是ISP模式
{
#if JVET_P1026_ISP_LFNST_COMBINATION
if ( m_pcEncCfg->getUseFastISP() )
{
m_modeCtrl->setISPWasTested(true);
}
#endif
tmpValidReturn = xIntraCodingLumaISP(*csTemp, subTuPartitioner, bestCurrentCost);//进行ISP模式的残差变换，量化，计算率失真代价等操作
if (csTemp->tus.size() == 0)
{
// no TUs were coded
csTemp->cost = MAX_DOUBLE;
continue;
}
#if JVET_P1026_ISP_LFNST_COMBINATION
// we save the data for future tests
m_ispTestedModes[m_curIspLfnstIdx].setModeResults((ISPType)cu.ispMode, (int)uiOrgMode.modeId, (int)csTemp->tus.size(), csTemp->cus[0]->firstTU->cbf[COMPONENT_Y] ? csTemp->cost : MAX_DOUBLE, csBest->cost);
csTemp->cost = !tmpValidReturn ? MAX_DOUBLE : csTemp->cost;
#else
if (!cu.mtsFlag && !cu.lfnstIdx)
{
// we save the data for future tests
m_ispTestedModes.setModeResults((ISPType)cu.ispMode, (int)uiOrgMode.modeId, (int)csTemp->tus.size(), csTemp->cus[0]->firstTU->cbf[COMPONENT_Y] ? csTemp->cost : MAX_DOUBLE, csBest->cost);
}
#endif
}//if(cu.ispMode)
else
{//否则进行普通的角度预测残差计算
tmpValidReturn = xRecurIntraCodingLumaQT( *csTemp, partitioner, uiBestPUMode.ispMod ? bestCurrentCost : MAX_DOUBLE, -1, TU_NO_ISP, uiBestPUMode.ispMod,
mtsCheckRangeFlag, mtsFirstCheckId, mtsLastCheckId, moreProbMTSIdxFirst );
}
if (!cu.ispMode && !cu.mtsFlag && !cu.lfnstIdx && !cu.bdpcmMode && !pu.multiRefIdx && !cu.mipFlag && testISP)
{
#if JVET_P0803_COMBINED_MIP_CLEANUP
m_regIntraRDListWithCosts.push_back( ModeInfoWithCost( cu.mipFlag, pu.mipTransposedFlag, pu.multiRefIdx, cu.ispMode, uiOrgMode.modeId, csTemp->cost ) );
#else
m_regIntraRDListWithCosts.push_back(ModeInfoWithCost(cu.mipFlag, pu.multiRefIdx, cu.ispMode, uiOrgMode.modeId, csTemp->cost));
#endif
}
//如果当前CU是ISP模式，且第一块TU不是亮度TU
if( cu.ispMode && !csTemp->cus[0]->firstTU->cbf[COMPONENT_Y] )
{
csTemp->cost = MAX_DOUBLE;
csTemp->costDbOffset = 0;
tmpValidReturn = false;
}
validReturn |= tmpValidReturn;
if( sps.getUseLFNST() && mtsUsageFlag == 1 && !cu.ispMode && mode >= 0 )
{
#if JVET_P0803_COMBINED_MIP_CLEANUP
m_modeCostStore[lfnstIdx][mode] = tmpValidReturn ? csTemp->cost : (MAX_DOUBLE / 2.0); //(MAX_DOUBLE / 2.0) ??
#else
m_modeCostStore[ lfnstIdx ][ testMip ? rdModeIdxList[ mode ] : mode ] = tmpValidReturn ? csTemp->cost : ( MAX_DOUBLE / 2.0 ); //(MAX_DOUBLE / 2.0) ??
#endif
}
DTRACE(g_trace_ctx, D_INTRA_COST, "IntraCost T [x=%d,y=%d,w=%d,h=%d] %f (%d,%d,%d,%d,%d,%d) \n", cu.blocks[0].x,
cu.blocks[0].y, (int)width, (int)height, csTemp->cost, uiOrgMode.modeId, uiOrgMode.ispMod,
pu.multiRefIdx, cu.mipFlag, cu.lfnstIdx, cu.mtsFlag);
if( tmpValidReturn )//如果是有效返回值，比较最优模式代价和当前模式代价
{
// check r-d cost
if( csTemp->cost < csBest->cost )//如果当前模式的cost小于目前最优的模式的cost
{
std::swap( csTemp, csBest );//当前模式成为最新的最优模式
uiBestPUMode = uiOrgMode;
bestBDPCMMode = cu.bdpcmMode;
if( sps.getUseLFNST() && mtsUsageFlag == 1 && !cu.ispMode )
{
m_bestModeCostStore[ lfnstIdx ] = csBest->cost; //cs.cost;
m_bestModeCostValid[ lfnstIdx ] = true;
}
if( csBest->cost < bestCurrentCost )
{
bestCurrentCost = csBest->cost;
}
#if JVET_P1026_ISP_LFNST_COMBINATION
if ( cu.ispMode )
{
m_modeCtrl->setIspCost(csBest->cost);
bestLfnstIdx = cu.lfnstIdx;
}
else if ( testISP )
{
m_modeCtrl->setMtsFirstPassNoIspCost(csBest->cost);
}
#else
if( !cu.ispMode && !cu.mtsFlag )
{
m_modeCtrl->setMtsFirstPassNoIspCost( csBest->cost );
}
#endif
}
if( !cu.ispMode && !cu.bdpcmMode && csBest->cost < bestCostNonBDPCM )
{
bestCostNonBDPCM = csBest->cost;
}
}//if( tmpValidReturn )
csTemp->releaseIntermediateData(); // 清空CU, PU, TU
if( m_pcEncCfg->getFastLocalDualTreeMode() )
{
if( cu.isConsIntra() && !cu.slice->isIntra() && csBest->cost != MAX_DOUBLE && costInterCU != COST_UNKNOWN && mode >= 0 )
{
if( m_pcEncCfg->getFastLocalDualTreeMode() == 2 )
{
//Note: only try one intra mode, which is especially useful to reduce EncT for LDB case (around 4%)
//注意：仅尝试一种帧内模式，这对于减少LDB情况下的EncT尤其有用（大约4％）
break;
}
else
{
if( csBest->cost > costInterCU * 1.5 )
{
break;
}
}
}
}
} // Mode loop
/*********RD候选模式的循环RDCost遍历结束,选出最优的模式**************/
cu.ispMode = uiBestPUMode.ispMod;//设置最优的ISP模式
#if JVET_P1026_ISP_LFNST_COMBINATION
cu.lfnstIdx = bestLfnstIdx;
#endif
if( validReturn )
{
cs.useSubStructure( *csBest, partitioner.chType, pu.singleChan( CHANNEL_TYPE_LUMA ), true, true, keepResi, keepResi );
}
csBest->releaseIntermediateData();
if( validReturn )
{
//=== update PU data ====
//更新PU数据
cu.mipFlag = uiBestPUMode.mipFlg;
#if JVET_P0803_COMBINED_MIP_CLEANUP
pu.mipTransposedFlag = uiBestPUMode.mipTrFlg;
#endif
pu.multiRefIdx = uiBestPUMode.mRefId;
pu.intraDir[ CHANNEL_TYPE_LUMA ] = uiBestPUMode.modeId;
cu.bdpcmMode = bestBDPCMMode;
}
}//代码块
//===== reset context models =====
m_CABACEstimator->getCtx() = ctxStart;
return validReturn;
}感谢H师兄的博客（狗头保命），多亏了师兄的博客我才能勉强看懂这个函数（顺带吐槽一下写VTM代码的人，真是连注释都不给，变量作用全靠猜）。
