About the author

Former technical expert of Didi Chuxing, currently in charge of mongodb, the OPPO document database, responsible for the R&D and operation and maintenance of the mongodb kernel of oppo's 10 million-level peak TPS/10 trillion-level data volume document database, has been focusing on distributed cache and high-performance server , database, middleware and other related research and development. Follow-up will continue to share "MongoDB kernel source code design, performance optimization, best operation and maintenance practices", Github account address: https://github.com/y123456yz

1. Background

mongodb is a full-featured distributed document database with natural advantages in high performance, dynamic expansion and contraction, high availability, easy deployment, ease of use, and massive data storage. Although mongodb has many advantages, there are many misunderstandings in China, and its influence needs to be improved.

1.1 MongoDB database score ranking over the years

DB-Engines is a website that ranks the popularity of database management systems. In recent years, MongoDB has ranked among the top five in the DB-Engines database popularity ranking. It can be seen from the DB-Engines ranking that the score data of mongodb over the years has shown a trend of continuous growth, as shown in the following figure:

The scores for the past two years are as follows:

It can be seen from the score rankings over the years that the overall trend of mongodb has shown a continuous growth trend, which has consolidated its advantages in the use of database systems.

1.2 mongodb roadmap: more than just a document database

The distributed transaction function has been supported since the mongodb-4.2 version. The current external document version has been iterated to version-4.2.11, and the distributed transaction function has been further enhanced. In addition, it can be seen from the product planning roadmap of mongodb-4.4 that mongodb officials will continue to invest in the development of query capabilities and usability enhancements , such as union multi-table joint query, index hiding, etc. The mongodb product roadmap is as follows:

It can be seen from the distributed transaction function support of mongodb-4.2 version and the roadmap plan of version 4.4 that the subsequent development of mongodb will continue to move in the direction of NewSQL, and it is expected to provide users with more comprehensive database service functions.

1.3 Talk about the current misunderstanding of mongodb in China? -Real feelings during service access

The mongodb ranking in Chapter 1 and the mongodb-4.x version product planning roadmap show that mongodb has an important position in the global database service system. At present, mongodb also provides important database storage services for our Internet system, supporting nearly 10 million QPS peak reading and writing every day, and trillions of data storage services.

However, in the process of real business access, before it was found that the business used mongodb for the first time, the most frequently consulted questions were as follows (through communication with the corresponding business development, we learned that these misunderstandings are all from the Internet, or heard from other peers ):

Question 1: Will mongodb lose data? It is said on the Internet that mongodb will lose data?

Answer: mongodb has a perfect rollback and write strategy (WriteConcern) mechanism, unless it is very extreme (for example, the master and slave are all hung up at the same time), data may be lost, and this problem will not exist in other cases.

During the promotion of mongodb by our company, I received three reports of data loss in mongodb in the business I contacted. Finally, through our traffic audit function, it was finally analyzed that it was caused by the logic of deleting data in the business.

Question 2: mongodb is not safe, and it is said on the Internet that the database is threatened by hacker attacks?

A: This is the fault of the DBA, because the attacked mongodb database has no authentication and no whitelisting functions, and hackers can attack with the IP. However, all online clusters of our company have authentication account authentication and whitelist functions, so there is no such problem.

Question 3: Many DBAs who operate and maintain Mysql react that mongodb is difficult to maintain?

A: In response to this issue, I have contacted some DBAs in the company and the former owner, and some people really feel this way. The main reason for the difficulty in maintenance is that mysql is a non-distributed database, the master-slave structure is deployed, and the number of nodes in a single cluster is limited. It is indeed faster and simpler to analyze problems than a distributed database. mongodb is a distributed database. Some large data and high traffic clusters have dozens of shards, and each shard has several master-slave nodes. There is a stateful logical relationship between shards and shards, so compared to mysql, it does maintain It will be more difficult.

However, from my own experience in maintaining mongodb clusters (thousands of online instances, the largest cluster data volume of trillions), as long as I master the principle of mongod sharded cluster implementation, I personally don't feel how difficult it is to operate and maintain.

Summary of the root causes of domestic misunderstandings:

Mongodb itself is very good, but many DBAs or corresponding developers can't control it.
The domestic systematic analysis of the implementation principle of the mongodb kernel is less, and there is no way to start when encountering problems.
In the network society, the problem of DBA or development itself has evolved into a problem of mongodb due to rumors.

1.4 What can I do to deal with the misunderstanding of mongodb in China? How to do?

The problems encountered by mongodb in our company should not be an isolated case. It is expected that other domestic companies will have similar situations. As I said in my Qcon speech, as long as more people in China join the research on mongodb implementation principles and more cases, Stepping on the pit and other sharing, the domestic influence of mongodb will become higher and higher, and the misunderstanding of mongodb in China will gradually disappear.

In the follow-up, I will continue to share the series of articles on "MongoDB kernel source code design, performance optimization, and best operation and maintenance practices". The series of articles have been shared on infoq, Zhihu, and github:

2. 2020 Summary

2020 is about to come to an end, this year is generally a year of harvest, and it is also the most fulfilling year in all previous jobs, cheer for yourself.

2.1 A year of harvest: praise yourself

The biggest gain in 2020 is that in the case of insufficient manpower, we have turned our company's unpopular mongodb database from the edge of internal elimination to the company's mainstream database by myself. At the same time, it has also won more external sharing and external awards, mainly as follows:

Internal mongodb promotion: from obsolete edge to company mainstream database

The first prize winner of the first mongodb Chinese community essay

The first prize of the Chinese community: the performance of a million-level high-concurrency MongoDB cluster has been improved by dozens of times and optimized

The second mongodb Chinese community essay first prize

MongoDB network transmission processing source code implementation and performance tuning - experience the ultimate design of kernel performance

QCON global software development conference sharing

Qcon sharing: OPPO trillion-level document database MongoDB cluster performance optimization practice

January 2020 mongodb year-end event invited to share

mongodb year-end event was invited to share: OPPO million-level high-concurrency MongoDB cluster performance dozens of times improved optimization practice

January 2021 mongodb year-end event invited to share

2021 mongodb year-end event sharing: performance optimization practice of trillion-level document database cluster

dbaplus is invited to share (2-3 sessions, time to be determined)

The optimization practice of improving the performance of a million-level high-concurrency mongodb cluster by dozens of times
Trillion-level document database mongodb cluster performance optimization practice

2.2 A year of stepping on the pit: reflect at all times and change the way of working

Although in the past year, our company's mongodb has been eliminated from the internal edge to the company's internal mainstream database, but in the process, I still stepped on a lot of pits, pitted myself, and pitted the business, haha, I have to give the corresponding business here. Not to accompany.

The process of stepping on the pit mainly focuses on the following aspects:

Unreasonable configuration and use of business clients
Monitoring is not in place
Unreasonable operation and maintenance, misoperation
Kernel stepping on the pit
......

In the process of accessing hundreds of services, there have been many pitfalls, and even the same core service has been shaken multiple times, and a core service avalanche fault has occurred. A typical case of avalanche failure was shared before, as follows:

Remember a mongodb core cluster avalanche failure

This case is one of dozens of trampling cases, and the remaining dozens of typical pit tramping cases will be sorted out and shared with the community in the spare time. Although stepping on the pit caused some unpleasantness, it can give other students some lessons to reduce other students stepping on similar pits.

It's not terrible to step on the pit, learn in the process of stepping on the pit, and reduce the step on the pit during the learning process

3. 21-year plan: continue to do something for the promotion and influence of mongodb in China

Domestic Internet companies that really have self-developed distributed systems are mainly concentrated in the head giants (for example: Ali, Tencent, etc.), and most companies still endure the pain points of Mysql sub-database and sub-table. In fact, with the support of mongodb-4.x distributed transaction, joint query and other functions, a large part of the scenarios can consider using the distributed document database mongodb to meet business needs and achieve rapid iterative development.

However, due to the lack of systematic mongodb data documents in China (mainly relying on official documents to meet the requirements) and the misunderstanding of mongodb on the Internet, the influence of mongodb in China is insufficient. In addition, there is hardly a complete book explaining the details of mongodb source-level implementation. In 2021, I hope to do something for the promotion of the document database mongodb in China, and set a small goal for myself here.

3.1 Short-term goals

The short-term goal is to share the core pit-stepping process, performance optimization cases, and computer room multi-activity cases encountered by our company in the process of accessing mongodb services with trillions of data to the community, so as to avoid other mongodb users in China from stepping on the same pit. These core stepping points are mainly as follows:

The bloody case caused by 300 data operations - record the unavailability of a 1 billion-level mongodb core cluster caused by 300 online data change operations.
Persistent OOM of the master node - record the core mongodb cluster failure caused by the unreasonable use of a business sorting operation.
Optimal index addition method for hundreds of billions/trillion-level clusters - record the cluster jitter and rapid recovery process caused by adding a background index of billions of core clusters at a time.
How to quickly restore the cluster when the number of links is exhausted - record a quick self-rescue process when the number of links in a tens of billions of core clusters is exhausted.
MongoDB computer room multi-active solution - realizes the "three harvests" of cost, performance, and consistency.
MongoDB thread model bottleneck and its optimization method.
Parallel migration - optimization practice of N-fold increase in cluster expansion rate.
The performance of mongodb cluster with hundreds of billions of core metadata has been improved several times.
The performance of the mongodb cluster with trillion-level data volume has been improved by dozens of times.
Cost savings - remember that a service has hundreds of billions of data migration mongodb, and hundreds of SSD servers save optimization practices.
How to quickly migrate hundreds of billions of data from an SSD high IO server to a SATA disk low IO server?
。。。。。。

Expected completion time: April-May 2021

3.2 Medium-term goals

Complete the sharing of core articles in the column " mongodb kernel source code implementation, performance tuning, and best operation and maintenance practices series " in Qcon, Zhihu, github, and itpub .

Complete the Chinese annotation analysis of the mongodb kernel source code in github: Chinese annotation analysis of the mongodb kernel source code

Expected completion date: November 2021

3.3 Annual goals

It is expected that through the sharing and sorting of the inside information of the core core technology of mongodb, the compilation and sorting of the following books will be completed:

"Distributed mongodb database kernel design and implementation"
"Best Operation and Maintenance Practice of Distributed MongoDB Database"

Expected completion time: end of December 2021

3.4 Internal company goals

The company's self-developed mongodb kernel is deeply optimized to maximize the solution to the jitter problem in the case of high concurrency, large traffic, and large data volume.
Quickly grasp the principle of mongodb-4.2 distributed transaction implementation, and start to vigorously promote the distributed transaction function.
Continue to share the cases of cluster stepping and performance optimization in the internal promotion process to the community.
other

Expected completion time: All year

2021 is coming, a new starting point, start first, come on! ! !

Tenfold improvement in performance of trillion-level database MongoDB cluster and multi-active disaster recovery practice in computer room

Inventory 2020-I want to do something for the promotion and promotion of the distributed document database mongodb in China