background
Because it is a short message business, as the company's business increases, the pressure on the database is also increasing. The current data volume is about 7 billion. Mongo only stores data within a week, and the data volume is nearly 100 million. I wanted to save costs before, so it has always been a single instance. With the increasing number of access customers, the number of text messages sent every day has also begun to increase significantly. The most worrying thing is to spend the holidays. I have been worried about if Mongo suddenly hangs up, haha! That is death-oriented programming.
Tangled
After several discussions, it was finally decided to use Mongo replica set read-write separation processing, so the service transformation began.
ready
- Several servers (192.168.0.101, 192.168.0.102, 192.168.0.103, 192.168.0.104) have been deployed on Mongo machines, install Mongo reference: Linux install Mongo
- Machine description: Two Mongo are slave libraries, one is only responsible for voting nodes and not processing data nodes (192.168.0.104)
upgrade
Key file
Through keyfile authentication, each mongod instance in the replica set uses the contents of the key file as a shared password to authenticate other members in the deployment. Only mongod instances with the correct key file can join the replica set. The content length of the key file must be between 6 and 1024 characters, and all members of the replica set must be the same.
openssl rand -base64 756 > /key/mongodb-keyfile
chmod 400 /key/mongodb-keyfileCopy the key file to each server that hosts the members of the replica set. Make sure that the user running the mongod instance is the owner of the file and can access the key file.
Modify the Mongo configuration of each service
First stop single instance Mongo (command: systemctl stop mongod), modify the net, security, and replication configuration information in the mongo configuration file /etc/mongod.conf
net:
port: 27017
bindIp: 0.0.0.0 # 允许所有ip远程连接Mongo
security:
authorization: enabled # 用户只能访问已为其授予特权的数据库资源和操作。
keyFile: /key/mongodb-keyfile # 秘钥文件路径
replication:
replSetName: "S1" # 副本集名称
enableMajorityReadConcern: false # 以为有投票节点，禁用该功能以防止存储高速缓存压力使部署无法固定。
Start Mongo
Use the command (systemctl start mongo) to enter mongo (Mongo that hopes to be the master node), and execute the processed replica set configuration
rs.initiate( {
_id : "s1",
members: [
{ _id: 0, host: "192.168.0.101:27017" },
{ _id: 1, host: "192.168.0.102:27017" },
{ _id: 2, host: "192.168.0.103:27017" },
{ _id: 3, host: "192.168.0.104:27017",arbiterOnly:true },
]
})Check the replica set configuration, the upgrade is complete (remote connection check)
rs.conf();Add temporary members to the replica set
rs.add( { host: "192.168.0.105:27017", priority: 0, votes: 0 } )Note:
When the votes and priority settings of the newly added secondary node are greater than zero, during its initial synchronization, even if the secondary node cannot provide read or become the primary node, because its data is still inconsistent, the secondary node is still counted as a member with voting rights .
This may result in a situation where most voting members are online but unable to elect the main members. To avoid this situation, please consider using priority:0 and votes:0 to add new secondary servers first. Then, once the member has transitioned to SECONDARY state, use rs.reconfig() to update its priority and vote.
If the configuration document returned by rs.conf() is the fifth element in the members array to update its priority and vote for it , use the following sequence of operations192.168.0.105:270171
var cfg = rs.conf();
cfg.members[4].priority = 1
cfg.members[4].votes = 1
rs.reconfig(cfg)Abnormal problem
1. Secret key file read authorization failed abnormally, causing startup failure
{"t":{"$date":"2020-10-17T14:38:26.085+08:00"},"s":"I", "c":"ACCESS", "id":20254, "ctx":"main","msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"Error reading file /key/mongodb-keyfile: Permission denied"}}}
solution:
I originally thought about whether the permissions were too small, so I directly authorized the 777 permissions and found that the file Permission too open error was directly reported, so the permission program was directly expanded.
1. By default, MongoDB mongodruns under a user account. After creation, set the owner and group of the directory to mongod:
sudo chown -R mongod:mongod /key/mongodb-keyfile2. According to the official documentation, you can read the file successfully as long as you modify the file group, but unfortunately it still fails to read the file. Through various search materials, the final solution can be found in the Rad Hat linux official directory permission file. Change SELinux policy configuration
chcon system_u:object_r:mongod_var_lib_t:s0 /key/mongodb-keyfile2. too open error reading the secret key file
solution:
1. Because on UNIX systems, the key file must not have group or world permissions. On Windows systems, the key file permissions are not checked. So only need to open the read permission, the permission is not allowed to be too large
chmod 400 /key/mongodb-keyfileReference documents:
https://docs.mongodb.com/manual/
https://stackoverflow.com/questions/28251531/permission-denied-to-read-file-owned-by-user