The key frames of the LocalMapping thread are passed from the Tracking thread. This thread will complete the three-phase initialization of the IMU and process the key frames in the queue (calculate BOW, update observations, descriptors, common views, insert into the map, etc.). Eliminate map points with poor quality based on the observation of the map points, and then generate new map points to make tracking more stable. Finally, it will detect whether the common view key frame of the current key frame is redundant, and finally add the current key frame to the closed-loop detection . in queue .
Local map points: In this thread, they are the map points of the current keyframe and the map points of the current keyframe. In fact, the local map points should be the map points in the map that can be observed by the current keyframe (displayed in red). point).
1. First set mbFinished to false, enter the while(1) loop, and then set mbAcceptKeyFrames to false, which means that LocalMapping is busy. Tracking, please do not send me key frames to disturb me. Always check whether mlNewKeyFrames is empty. The condition for ending the while(1) loop is: mbFinishRequested is true;
2. When mlNewKeyFrames is not empty and mbBadImu is false, process the key frames in the queue.
ProcessNewKeyFrame()：
1) Take out the front key frame in the queue, record it as the current key frame (mpCurrentKeyFrame), and then delete the key frame in the original queue;
2) Calculating the Bow of the current key frame is actually updating the mBowVec and mFeatVec containers of the current key frame. mBowVec is used to calculate the similarity score of the two key frames, and mFeatVec is used to accelerate feature point matching. Both are prepared for LocalClosing;
mBowVec：<单词id，weight>如果图像中多个特征点对应同一个单词，weight会叠加。
mFeatVec：<节点id（L=2时），该节点id下面的单词对应的特征点在当前关键帧中的编号（存在容器中）>3) Process the valid map point in the current key frame. If the map point is not an observation from the current frame, add an observation to the map point (update mObservations and nObs), update the average observation direction, observation distance range and best of the map point. Descriptor. If the map point is an observation from the current key frame, put the map point into mlpRecentAddedMapPoints and wait for subsequent verification by the MapPointCulling function;
std::map<KeyFrame*,std::tuple<int,int> > mObservations;//地图点的成员变量<能观测到该地图点的关键帧指针，该地图点在该关键帧中对应特征点的索引>
std::vector<MapPoint*> mvpMapPoints;//关键帧的地图点，索引为特征点编号，成功生成地图点为该地图点的指针，否则为NULL。Note: nObs indicates how many cameras (key frames) can observe the map point. Binocular and RGB-D add 2 each time, and monocular adds 1.
4) Update the connection relationship between key frames: UpdateConnections() is to update mConnectedKeyFrameWeights = KFcounter, mvpOrderedConnectedKeyFrames: The common view key frame pointers of the current key frame are sorted from large to small according to the number of common view map points. Finally, updating the connection of the spanning tree is to find the parent keyframe of the current keyframe (the keyframe with the highest degree of common vision), establish a two-way connection relationship, and use the current keyframe as its child keyframe, and all keyframes have only one parent keyframe. frame (initialized reference keyframe has no parent keyframe), and finally inserts the current keyframe into the map (mspKeyFrames);
map<KeyFrame *, int> KFcounter;//<关键帧指针，当前关键帧与关键帧共有的地图点数>
vector<pair<int, KeyFrame *>> vPairs;//<当前关键帧与关键帧共有的地图点数,关键帧指针>,
//存储的是共有的地图点数 >= th（15）的关键帧，或者当前关键帧与所有关键帧共有的地图点数都小于th则会把权重最大的加入。3. MapPointCulling(): Detect the map points in mlpRecentAddedMapPoints and set different observation thresholds (cnThObs) according to the camera type: 2 for single eye and 3 for the rest;
1) Recall rate = actual number of frames observed for the map point (mnFound)/number of frames theoretically observed for the map point (mnVisible) < 0.25, the map point is considered a bad point;
2) Current keyframe id - If the id of the frame where the map point was first created is >= 2 and the observation of the map point (nObs) <= cnThObs, the map point is considered a bad point;
3) If 3 key frames have passed since the map point was established without being eliminated, it is considered to be a high-quality map point.
4. Use the current keyframe and adjacent keyframes to generate new map points through triangulation, making tracking more stable. CreateNewMapPoints(): Find the 30 keyframes with the highest common view of the current keyframes. If it is IMU mode, more keyframes will be added, and these keyframes will be stored in vpNeighKFs. If mlNewKeyFrames is empty at this time, new map points will be generated through triangulation. If mlNewKeyFrames is not empty, return and process the new keyframe first. The newly generated map points are stored in mspMapPoints (which stores all map points in the map), and finally the newly generated points are put into the detection queue mlpRecentAddedMapPoints.
5. Set mbAbortBA to false . If the last keyframe in the queue has been processed, check and fuse the repeated map points in the current keyframe and adjacent keyframes (two-level adjacent). In the process, Check whether mbAbortBA is true, if true, stop the process; SearchInNeighbors():
1) The 30 key frames with the highest degree of co-viewing with the current key frame are called first-level adjacent key frames. The 20 keyframes with the highest co-viewing degree of the first-level adjacent keyframes are used as the second-level adjacent keyframes. Some prevKFs are added in IMU mode, and these keyframes are stored in vpTargetKFs;
2) Perform forward fusion of the map points of the current key frame and the map points of the key frames in vpTargetKFs (fuse the map points in the current key frame into the key frames in vpTargetKF) and reverse fusion;
3) The integration strategy is:
a. If there is no map point at the corresponding position of the back projection of the map point, add the observation directly;
b. If there is a map point at the corresponding position of the back projection of the map point, merge the two map points into the map point with more observations.
4) Update the descriptor, depth, main direction of observation and other attributes of the current keyframe map point, and finally update the connection relationship between keyframes.
6. The last keyframe in the queue has been processed, and the loop closure detection does not request to stop LocalMapping. At the same time, the number of keyframes in the current map is greater than 2 and it is not in IMU mode or the map where the current keyframe is located has not completed the first IMU initialization. , perform local BA optimization. First-level common view keyframe: The common view keyframe pose of the current keyframe (will be optimized). Keyframes that can be observed at local map points but do not belong to the first-level common view keyframe will be added to the optimization map but the pose will not. It is optimized, local map points are added as vertices, edges are reprojections of local map points, and finally the first-level common view keyframe pose and local map point coordinates are updated;
Note: The second parameter is passed by address. When the mbAbortBA status here changes, BA can be executed/stopped in time.
Optimizer::LocalInertialBA(mpCurrentKeyFrame, &mbAbortBA, mpCurrentKeyFrame->GetMap(), num_FixedKF_BA, num_OptKF_BA, num_MPs_BA, num_edges_BA, bLarge, !mpCurrentKeyFrame->GetMap()->GetIniertialBA2());7. Detect and eliminate redundant key frames in key frames adjacent to the current key frame. Redundant key frame judgment criteria: 90% of the map points of this key frame can be observed by other key frames, but the newly added key frame will not be eliminated;
KeyFrameCulling()：
1) Update mvpOrderedConnectedKeyFrames of the current key frame, take out the common view key frame (mvpOrderedConnectedKeyFrames) of the current key frame and store it in vpLocalKeyFrames. When there are more than 22 key frames in front of the current key frame, last_ID records the id of the 22nd last key frame in the map. ,count=21;
2) Traverse all common view key frames (vpLocalKeyFrames), extract the map points of each common view key frame, update nRedundantObservations and nMPs, and finally determine whether the key frame is redundant. If it is redundant and not in IMU mode, delete the key frame;
3) Conditions for not deleting this key frame in redundant and IMU mode:
a. There are less than 21 keyframes in the map;
b. The difference between the key frame and the current key frame ID is 1 (the latest key frame added);
c. The following two conditions are not met:
if((bInitImu && (pKF->mnId < last_ID) && t < 3.) || (t < 0.5))
else if(!mpCurrentKeyFrame->GetMap()->GetIniertialBA2() && ((pKF->GetImuPosition() - pKF->mPrevKF->GetImuPosition()).norm() < 0.02) && (t < 3))4) pKF->SetBadFlag()：
a. The initial keyframe cannot be deleted. mbNotErase (which is set in LoopClosing) indicates that it should not be deleted, so mbToBeErased is set to true and pretends to have been deleted, but in fact it is not deleted.
b. Delete the common view keyframes and map points of the current keyframe;
c. Update the spanning tree.
8. Add the current key frame to the closed-loop detection queue.