Sharded cluster-build

Sharded cluster

Database applications with high data volume and throughput will put great pressure on the performance of a single machine. A large query volume will exhaust the CPU of a single machine. A large amount of data will put great pressure on the storage of a single machine, and will eventually exhaust the system's memory. Instead, the pressure is transferred to disk IO.

To solve these problems, there are two basic methods: vertical expansion and horizontal expansion.

Vertical expansion: Add more CPU and storage resources to expand capacity.

Horizontal scaling: Distribute data sets across multiple servers. Horizontal expansion is sharding.

Principle introduction

Sharding is the method MongoDB uses to split large collections onto different servers (or a cluster).

For example, if the database has a 1TB dataset and has 4 shards, then each shard may only hold 256 GB of data. If there are 40 shards, then each shard may only have 25GB of data.

The sharding of data in MongoDB takes sets as the basic unit, and the data in the set is divided into multiple parts through shard keys. In fact, the shard key is to select a key in the collection and use the value of the key as the basis for data splitting.

Generally, shard keys are sharded using range or hash methods.

Cluster construction

Environmental preparation

First create a directory as follows:

• Create the mongodb directory in the root directory. Folders (node1, node2, node3) are created internally in sequence.
• Create: config-server1, mongos1, shard11, shard21 folders under the node1 folder
  • Create backup, config, db folders in the config-server1 directory
  • mongos1, shard11, shard21 are created as above
• Created under node2 and node3 folders

Configuration service

Create container

Enter the config-serverx directory of node1, node2, and node3. Execute the docker command to create and start the docker container

Configure service 1

#进入目录
cd /root/mongodb/node1/config-server1
#创建并启动容器。
docker run --restart=always --privileged=true -p 10021:27019 -v $PWD/config:/etc/mongod -v $PWD/db:/data/db -d --name pro-file-server-config1 mongo:4.0.0 -f /etc/mongod/config.conf --configsvr --replSet "rs-file-server-config-server" --bind_ip_all

Configure service 2

#进入目录
cd /root/mongodb/node2/config-server2
#创建并启动容器。
docker run --restart=always --privileged=true -p 10022:27019 -v $PWD/config:/etc/mongod -v $PWD/db:/data/db -d --name pro-file-server-config2 mongo:4.0.0 -f /etc/mongod/config.conf --configsvr --replSet "rs-file-server-config-server" --bind_ip_all

Configure service 2

#进入目录
cd /root/mongodb/node3/config-server3
#创建并启动容器。
docker run --restart=always --privileged=true -p 10023:27019 -v $PWD/config:/etc/mongod -v $PWD/db:/data/db -d --name pro-file-server-config3 mongo:4.0.0 -f /etc/mongod/config.conf --configsvr --replSet "rs-file-server-config-server" --bind_ip_all`

Configure Mongo

Associate 3 configuration services together

Enter any config-server through docker to configure the connection between the three servers

#进入pro-file-server-config1配置服务器
docker exec -it pro-file-server-config1 /bin/bash
#连接mongo 配置服务器暴露的默认端口为27019
mongo -port 27019
#执行配置内容
rs.initiate({
    
    
_id: "rs-file-server-config-server",
configsvr: true,
members: [
{
    
     _id : 0,host : "192.168.136.163:10021" },
{
    
     _id : 1,host : "192.168.136.163:10022" },
{
    
     _id : 2, host : "192.168.136.163:10023" }
]
});
#查看状态
rs.status()

routing service

Create container

mongos1

#进入目录
cd /root/mongodb/node1/mongos1
#创建并启动容器。
docker run --restart=always --privileged=true -p 10011:27017 -v $PWD/config:/etc/mongod -v $PWD/db:/data/db -d --entrypoint mongos --name pro-file-server-mongos1 mongo:4.0.0 -f /etc/mongod/config.conf --configdb rs-file-server-config-server/192.168.50.100:10021,192.168.50.100:10022,192.168.50.100:10023 --bind_ip_all

mongos2

#进入目录
cd /root/mongodb/node2/mongos2
#创建并启动容器。
docker run --restart=always --privileged=true -p 10012:27017 -v $PWD/config:/etc/mongod -v $PWD/db:/data/db -d --entrypoint mongos --name pro-file-server-mongos2 mongo:4.0.0 -f /etc/mongod/config.conf --configdb rs-file-server-config-server/192.168.50.100:10021,192.168.50.100:10022,192.168.50.100:10023 --bind_ip_all

mongos3

#进入目录
cd /root/mongodb/node3/mongos3
#创建并启动容器。
docker run --restart=always --privileged=true -p 10013:27017 -v $PWD/config:/etc/mongod -v $PWD/db:/data/db -d --entrypoint mongos --name pro-file-server-mongos3 mongo:4.0.0 -f /etc/mongod/config.conf --configdb rs-file-server-config-server/192.168.50.100:10021,192.168.50.100:10022,192.168.50.100:10023 --bind_ip_all

Configure Mongo

Enter any mongos through docker to configure the connection between the three servers

#进入pro-file-server-mongos1服务器
docker exec -it pro-file-server-mongos1 /bin/bash
#连接mongo
mongo -port 27018
#执行配置内容
sh.addShard("rs-file-server-shard1-server/192.168.136.163:10031,192.168.136.163:10032,192.168.136.163:10033")
sh.addShard("rs-file-server-shard2-server192.168.136.163:10041,192.168.136.163:10042,192.168.136.163:10043")

Sharded service

Create container

Sharding service share11

cd /root/mongodb/node1/shard21
docker run --restart=always --privileged=true -p 10041:27018 -v $PWD/config:/etc/mongod -v $PWD/backup:/data/backup -v $PWD/db:/data/db -d --name pro-file-server-shard21 mongo:4.0.0 -f /etc/mongod/config.conf --shardsvr --replSet "rs-file-server-shard2-server" --bind_ip_all

Sharding service share12

cd /root/mongodb/node2/shard22
docker run --restart=always --privileged=true -p 10042:27018 -v $PWD/config:/etc/mongod -v $PWD/backup:/data/backup -v $PWD/db:/data/db -d --name pro-file-server-shard22 mongo:4.0.0 -f /etc/mongod/config.conf --shardsvr --replSet "rs-file-server-shard2-server" --bind_ip_all

Sharding service share13

cd /root/mongodb/node3/shard23
docker run --restart=always --privileged=true -p 10043:27018 -v $PWD/config:/etc/mongod -v $PWD/backup:/data/backup -v $PWD/db:/data/db -d --name pro-file-server-shard23 mongo:4.0.0 -f /etc/mongod/config.conf --shardsvr --replSet "rs-file-server-shard2-server" --bind_ip_all

Configure Mongo

Enter any share through docker to configure the connection between the three servers

#进入pro-file-server-shard21服务器
docker exec -it pro-file-server-shard21 /bin/bash
#连接mongo
mongo -port 27018
#执行配置内容
`rs.initiate({
     
     
_id: "rs-file-server-shard2-server",
members: [
{
     
      _id : 0, host : "192.168.136.163:10041" },
{
     
      _id : 1, host : "192.168.136.163:10042" },
{
     
      _id : 2, host : "192.168.136.163:10043" }
]
});`
#查看状态
rs.status()

Sharding configuration

#mongodb客户端连接
mongo 192.168.136.163:10011

#创建分片数据库test
sh.enableSharding("test")

#将collection加入分片并设置分片字段
sh.shardCollection("test.user", {
    
    "_id": "hashed" })

Server description

routing service

IP	port
192.168.136.163	10011
192.168.136.163	10012
192.168.136.163	10013

Configuration service

IP	port
192.168.136.163	10021
192.168.136.163	10022
192.168.136.163	10023

Sharded service

Sharding service 01	Sharding service 02
192.168.136.163:10031	192.168.136.163:10041
192.168.136.163:10032	192.168.136.163:10042
192.168.136.163:10033	192.168.136.163:10043