1. The Jacobian matrix of DSO consists of three parts:
One part is geometric Jacobian, which describes the relative geometric quantities of each quantity, such as the rate of change of rotation and translation;
The second part is the luminosity Jacobian, which describes the Jacobian of each quantity relative to the luminosity parameter;
Three-part image Jacobian, that is, image gradient;
The objective function is the formula 4 in the paper on the function of the smallest luminosity error
2.FEJ
The geometric and luminosity Jacobian is usually a smooth function relative to the independent variables; while the image Jacobian depends on the image data and is not smooth enough; therefore, in the optimization process, the geometric and luminosity Jacobian is only calculated once at the beginning of the iteration. It will remain unchanged since then. The image Jacobian is updated with iteration. This approach is called First-Estimate-Jacobian (FEJ).
3. Diagonal array
The inverse depth point needs a host frame, and then the point needs to be projected onto a new frame, so one point corresponds to two frames, and one inverse depth information. So the pose part of the Hessian matrix is not a diagonal matrix. But the inverse depth point part is still a diagonal array .
4. Marginalization
From a probabilistic point of view, we call this step marginalization because we actually convert the problem of finding (∆xc, ∆xp) (pose, inverse depth) into a problem of finding ∆xc first and then ∆xp process. This step is equivalent to doing a conditional probability expansion:
P (xc, xp) = P (xc) · P (xp |xc ). The
result is that the marginal distribution of xc is obtained, so it is called marginalization. In the above-mentioned marginalization process, we actually marginalize all the inverse depth points. At the beginning, the inverse depth was eliminated first, so the inverse depth was marginalized, and the edge distribution of the pose was obtained.
For example, there are frame1, frame2, frame3...frame8 in the key frame window and the feature points pt1...ptn on these frames. For example, if a frame of framex is to be discarded and kicked out of the key frame window, the direct way to appear in the mind is to directly discard framex and related feature points pt, and only optimize the remaining frames in the window and their corresponding feature points. However, directly discarding the variables will result in the loss of information. Framex may be able to constrain adjacent frames more, and the direct discarding method will destroy these constraints. In DSO, the correct way to remove a framex from the window is to marginalize it from the hession. Marginalization is the use of Schur complement
Solve the equation as follows:
The marginalization we pay attention to. That is to say, we only solve the variables that we want to keep, and we don't care about the variables that we want to marginalize, so as to achieve the purpose of reducing calculations. Obviously, what you want to marginalize is the inverse depth information of the point, and what you want to keep is information such as pose and luminosity. What we want to marginalize here is to drop δρ (about the inverse depth), and keep δX (about the pose photometric camera parameters, etc.), and eliminate the above equation to get:
Among them 
is called Shure's supplement 
in the 
middle. With this upper triangular formula, we can directly calculate δX.
In this way, we can update some variables iteratively. In the above process, we constructed Hx=b using marginalized variable information, that is to say, we have no artificial discard constraints, so we will not lose information, but when calculating the results, we only update us The values of those variables that you want to keep. This is the process of marginalization. The marginalization in DSO means that we no longer need this point/frame. When it is marginalized, we pass its information to the later a priori instead of using this point/frame.
5. Sliding window
In Schur elimination, that is, Marginalization. Solving the incremental equation of the pose part is the main amount of calculation (visual slamP253). The physical meaning of the coefficient S of that equation: the non-zero matrix block on the off-diagonal line of the S matrix indicates that there is a common observation roadmap point between the two camera variables corresponding to this place, sometimes called common view ( Co-visibility). Conversely, if the block is zero, it means that the two cameras do not observe together.
6. Marginalized code
void EnergyFunctional::marginalizePointsF()
{
assert(EFDeltaValid);
assert(EFAdjointsValid);
assert(EFIndicesValid);
allPointsToMarg.clear();
//循环关键帧窗口的帧
for(EFFrame* f : frames)
{
//循环点
for(int i=0;i<(int)f->points.size();i++)
{
EFPoint* p = f->points[i];
//判定为需要边缘化的点
if(p->stateFlag == EFPointStatus::PS_MARGINALIZE)
{
p->priorF *= setting_idepthFixPriorMargFac;
for(EFResidual* r : p->residualsAll)
if(r->isActive())
connectivityMap[(((uint64_t)r->host->frameID) << 32) + ((uint64_t)r->target->frameID)][1]++;
allPointsToMarg.push_back(p);
}
}
}
accSSE_bot->setZero(nFrames);
accSSE_top_A->setZero(nFrames);
//准备边缘化的点算一些东西
for(EFPoint* p : allPointsToMarg)
{
//这些点用来计算的acc会保留下来
//这些点还会参与位姿 相机参数以及仿射的矩阵
//计算的结果存在Hdd_accLF 等 标号为LF变量中
accSSE_top_A->addPoint<2>(p, this);
//用LF来计算SCH的相关值吧
accSSE_bot->addPoint(p, false);
//算完了就把这个点删除了
removePoint(p);
}
MatXX M, Msc;
VecX Mb, Mbsc;
// 解出来一组M Mb Msc Mbsc 来 是解方程组的元素
accSSE_top_A->stitchDouble(M, Mb, this, false, false);
accSSE_bot->stitchDouble(Msc, Mbsc, this);
resInM+= accSSE_top_A->nres[0];
//这步跟方程结合看。就是计算了边缘化的点的对舒尔方程的约束
//点没了 约束留下来了
//舒尔消元的公式左侧第二行那个减式
MatXX H = M - Msc;
//舒尔消元的公式右侧第二行那个减式
VecX b = Mb - Mbsc;
if(setting_solverMode & SOLVER_ORTHOGONALIZE_POINTMARG)
{
// have a look if prior is there.
bool haveFirstFrame = false;
for(EFFrame* f : frames) if(f->frameID==0) haveFirstFrame=true;
if(!haveFirstFrame)
orthogonalize(&b, &H);
}
//将舒尔补乘个系数放到整体的黑塞矩阵中去，完成了点被边缘化掉但是
//关于计算相机位姿部分的约束保留下来的功效
HM += setting_margWeightFac*H;
bM += setting_margWeightFac*b;
//零空间 重新修正一次 bM HM
if(setting_solverMode & SOLVER_ORTHOGONALIZE_FULL)
{
orthogonalize(&bM, &HM);
}
EFIndicesValid = false;
makeIDX();
}