Mongodb of cluster structures and Troubleshooting

First, build a cluster Mongodb

1, network topology.

Mongodb of cluster structures and Troubleshooting

2, is mounted on the three nodes mongodb

Reference this blog installation documentation, extract it to the appropriate location, and configure the environment variables on the line, do not start.

3, respectively, create a file folder used to store data in / app / mongodb directory three servers

mkdir -p /app/mongodb/{master|slave|arbiter}

主节点配置文件:   #新建
#master.conf
dbpath=/app/mongodb/master
logpath=/app/mongodb/master.log
pidfilepath=/app/mongodb/master.pid
#keyFile=/app/mongodb/mongodb.key     (待群集配置好后,再启用该功能)
directoryperdb=true
logappend=true
replSet=testdb
bind_ip=10.3.152.78
port=27017
#auth=true                     (待群集配置好后,再启用该功能)
oplogSize=100
fork=true
noprealloc=true
maxConns=4000

从节点配置文件:
#slave.conf
dbpath=/app/mongodb/slave
logpath=/app/mongodb/slave.log
pidfilepath=/app/mongodb/slave.pid
#keyFile=/app/mongodb/mongodb.key
directoryperdb=true
logappend=true
replSet=testdb
bind_ip=10.3.151.34
port=27017
#auth=true
oplogSize=100
fork=true
noprealloc=true
maxConns=4000

仲裁节点配置文件:
#arbiter.conf
dbpath=/app/mongodb/arbiter
logpath=/app/mongodb/arbiter.log
pidfilepath=/app/mongodb/arbiter.pid
#keyFile=/app/mongodb/mongodb.key
directoryperdb=true
logappend=true
replSet=testdb
bind_ip=10.3.151.34
port=27020
#auth=true
oplogSize=100
fork=true
noprealloc=true
maxConns=4000

4, mounted on the main OpenSSL node, and generates KeyFile file for the encrypted communication between the three libraries

yum -y install openssl
openssl rand -base64 756 > /app/mongodb/mongodb.key
chmod 400 /app/mongodb/mongodb.key

5, the files are copied from the mongodb.key nodes and the arbitration server / app / mongodb / directory, and the same set of permissions

6, start mongodb three servers

主节点: mongod -f /etc/mongodb_master.conf
从节点:mongod -f /etc/mongodb_slave.conf
仲载点:mongod -f /etc/mongodb_arbiter.conf

7, on the primary node, the cluster configuration information is added

Mongodb of cluster structures and Troubleshooting

8, view the status of the current cluster

switched to db admin
testdb:PRIMARY> rs.status()
{
    "set" : "testdb",
    "date" : ISODate("2019-07-18T08:53:54.483Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "10.3.152.78:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",      #主节点
            "uptime" : 1302,
            "optime" : Timestamp(1563439850, 1),
            "optimeDate" : ISODate("2019-07-18T08:50:50Z"),
            "electionTime" : Timestamp(1563439854, 1),
            "electionDate" : ISODate("2019-07-18T08:50:54Z"),
            "configVersion" : 1,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "10.3.151.34:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",        #从节点
            "uptime" : 184,
            "optime" : Timestamp(1563439850, 1),
            "optimeDate" : ISODate("2019-07-18T08:50:50Z"),
            "lastHeartbeat" : ISODate("2019-07-18T08:53:54.360Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T08:53:54.360Z"),
            "pingMs" : 0,
            "lastHeartbeatMessage" : "could not find member to sync from",
            "configVersion" : 1
        },
        {
            "_id" : 2,
            "name" : "10.3.151.34:27020",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",       #仲载节点
            "uptime" : 184,
            "lastHeartbeat" : ISODate("2019-07-18T08:53:54.360Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T08:53:54.359Z"),
            "pingMs" : 0,
            "configVersion" : 1
        }
    ],
    "ok" : 1
}

9, adding super administrator privileges on the main library

testdb:PRIMARY> use admin
switched to db admin
testdb:PRIMARY> db.createUser(
... {
... user:"admin",
... pwd:"abc123",
... roles:["readWriteAnyDatabase","dbAdminAnyDatabase","userAdminAnyDatabase","clusterAdmin"]
... }
... )
Successfully added user: {
    "user" : "admin",
    "roles" : [
        "readWriteAnyDatabase",
        "dbAdminAnyDatabase",
        "userAdminAnyDatabase",
        "clusterAdmin"
    ]
}

10, this time for mongodb plus authentication

Note remove the three profiles
# = keyFile / App / MongoDB / mongodb.key
# to true the auth =

11, restart all mongodb, Denden master node server, view the current status of the cluster

[root@ops-site master]# mongo 10.3.152.78
MongoDB shell version: 3.0.15
connecting to: 10.3.152.78/test
testdb:PRIMARY> use admin
switched to db admin
testdb:PRIMARY> db.auth("admin","abc123")        #加密功能验证成功
1
testdb:PRIMARY> rs.status()              #当前状态正常
{
    "set" : "testdb",
    "date" : ISODate("2019-07-18T09:05:31.654Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "10.3.152.78:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 152,
            "optime" : Timestamp(1563440306, 4),
            "optimeDate" : ISODate("2019-07-18T08:58:26Z"),
            "electionTime" : Timestamp(1563440628, 1),
            "electionDate" : ISODate("2019-07-18T09:03:48Z"),
            "configVersion" : 1,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "10.3.151.34:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 103,
            "optime" : Timestamp(1563440306, 4),
            "optimeDate" : ISODate("2019-07-18T08:58:26Z"),
            "lastHeartbeat" : ISODate("2019-07-18T09:05:30.551Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T09:05:30.550Z"),
            "pingMs" : 0,
            "configVersion" : 1
        },
        {
            "_id" : 2,
            "name" : "10.3.151.34:27020",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 59,
            "lastHeartbeat" : ISODate("2019-07-18T09:05:30.540Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T09:05:31.002Z"),
            "pingMs" : 0,
            "configVersion" : 1
        }
    ],
    "ok" : 1
}

12, create a test database in the main library, insert python sets and several documents to see if sync files from the library

主库创建:
[root@ops-site master]# mongo 10.3.152.78
MongoDB shell version: 3.0.15
connecting to: 10.3.152.78/test
testdb:PRIMARY> use admin
switched to db admin
testdb:PRIMARY> db.auth("admin","abc123")
1
testdb:PRIMARY> use test
switched to db test
testdb:PRIMARY> db.createCollection("python")
{ "ok" : 1 }
testdb:PRIMARY> db.python.insert({x:1000})
WriteResult({ "nInserted" : 1 })
testdb:PRIMARY> db.python.insert({x1:1000})
WriteResult({ "nInserted" : 1 })
从库查看:(同步成功)
[root@nbutest arbiter]# mongo 10.3.151.34
MongoDB shell version: 3.0.15
connecting to: 10.3.151.34/test
testdb:SECONDARY> use admin
switched to db admin
testdb:SECONDARY> db.auth("admin","abc123")
1
testdb:SECONDARY> db.getMongo().setSlaveOk();       #设置从库可读(默认不能读 )
testdb:SECONDARY> use test
switched to db test
testdb:SECONDARY> db.python.find()
{ "_id" : ObjectId("5d303765712c279b0cae08b9"), "x" : 1000 }
{ "_id" : ObjectId("5d303769712c279b0cae08ba"), "x1" : 1000 }
testdb:SECONDARY> 

13, the main library stopped to see if become the main library from the library

主库:
[root@ops-site master]# mongo 10.3.152.78
MongoDB shell version: 3.0.15
connecting to: 10.3.152.78/test
testdb:PRIMARY> use admin
switched to db admin
testdb:PRIMARY> db.auth("admin","abc123")
1
testdb:PRIMARY> db.shutdownServer()
2019-07-18T17:15:07.766+0800 I NETWORK  DBClientCursor::init call() failed
server should be down...
2019-07-18T17:15:07.768+0800 I NETWORK  trying reconnect to 10.3.152.78:27017 (10.3.152.78) failed
2019-07-18T17:15:07.768+0800 I NETWORK  reconnect 10.3.152.78:27017 (10.3.152.78) ok
2019-07-18T17:15:07.770+0800 I NETWORK  DBClientCursor::init call() failed
2019-07-18T17:15:07.814+0800 I NETWORK  trying reconnect to 10.3.152.78:27017 (10.3.152.78) failed
2019-07-18T17:15:07.814+0800 I NETWORK  reconnect 10.3.152.78:27017 (10.3.152.78) ok
2019-07-18T17:15:08.475+0800 I NETWORK  Socket recv() errno:104 Connection reset by peer 10.3.152.78:27017
2019-07-18T17:15:08.475+0800 I NETWORK  SocketException: remote: 10.3.152.78:27017 error: 9001 socket exception [RECV_ERROR] server [10.3.152.78:27017] 

从库:
[root@nbutest arbiter]# mongo 10.3.151.34
MongoDB shell version: 3.0.15
connecting to: 10.3.151.34/test
testdb:PRIMARY>             #已经提示PRIMARY

14, starts the main library to see if the switch to the main library on 10.3.152.78

[root@ops-site master]# mongod -f /etc/mongodb_master.conf 
note: noprealloc may hurt performance in many applications
about to fork child process, waiting until server is ready for connections.
forked process: 28829
child process started successfully, parent exiting
You have mail in /var/spool/mail/root
[root@ops-site master]# mongo 10.3.152.78
MongoDB shell version: 3.0.15
connecting to: 10.3.152.78/test
testdb:PRIMARY>              #切换成功

Second, the failure of the cluster exclusion

1, from the state of the node processing method RECOVERING

1、登陆到/app/mongodb/slave目录上
[root@nbutest slave]# pwd
/app/mongodb/slave
[root@nbutest slave]# ll
总用量 24
drwxr-xr-x 3 root root 4096 7月  18 17:03 admin
drwxr-xr-x 2 root root 4096 7月  18 17:09 journal
drwxr-xr-x 3 root root 4096 7月  18 17:03 local
-rw-r--r-- 1 root root    6 7月  18 17:03 mongod.lock
-rw-r--r-- 1 root root   69 7月  18 16:34 storage.bson
drwxr-xr-x 3 root root 4096 7月  18 17:09 test

2、把该目录上的所有数据删除
如果稳妥一点,可以把配置文件的数据存放重新定一个新的位置 

3、启动mongodb,重新同步
testdb:PRIMARY> rs.status()
{
    "set" : "testdb",
    "date" : ISODate("2019-07-18T09:30:03.958Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "10.3.152.78:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 743,
            "optime" : Timestamp(1563441001, 1),
            "optimeDate" : ISODate("2019-07-18T09:10:01Z"),
            "electionTime" : Timestamp(1563441462, 1),
            "electionDate" : ISODate("2019-07-18T09:17:42Z"),
            "configVersion" : 1,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "10.3.151.34:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",          #已经恢复
            "uptime" : 130,
            "optime" : Timestamp(1563441001, 1),
            "optimeDate" : ISODate("2019-07-18T09:10:01Z"),
            "lastHeartbeat" : ISODate("2019-07-18T09:30:03.747Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T09:30:02.743Z"),
            "pingMs" : 0,
            "configVersion" : 1
        },
        {
            "_id" : 2,
            "name" : "10.3.151.34:27020",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 743,
            "lastHeartbeat" : ISODate("2019-07-18T09:30:03.730Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T09:30:03.846Z"),
            "pingMs" : 0,
            "configVersion" : 1
        }
    ],
    "ok" : 1
}

2, if you want to port node is modified from 27018

1、停掉从节点服务
2、修改配置文件的端口为28018
3、登陆主节点,删除当前从节点服务器
[root@ops-site master]# mongo 10.3.152.78
MongoDB shell version: 3.0.15
connecting to: 10.3.152.78/test
testdb:PRIMARY> use admin
switched to db admin
testdb:PRIMARY> db.auth("admin","abc123")
1
testdb:PRIMARY> rs.remove("10.3.151.34:27017")
{ "ok" : 1 }
testdb:PRIMARY> rs.add("10.3.151.34:27018")
{ "ok" : 1 }
testdb:PRIMARY> 

4、启动从节点
[root@nbutest slave]# mongod -f /etc/mongodb_slave.conf 
note: noprealloc may hurt performance in many applications
about to fork child process, waiting until server is ready for connections.
forked process: 25209
child process started successfully, parent exiting
[root@nbutest slave]# netstat -tlunp | grep mongod
tcp        0      0 10.3.151.34:27018           0.0.0.0:*                   LISTEN      25209/mongod        
tcp        0      0 10.3.151.34:27020           0.0.0.0:*                   LISTEN      24735/mongod 

5、查看群集的状态
testdb:PRIMARY> rs.status()
{
    "set" : "testdb",
    "date" : ISODate("2019-07-18T09:41:33.604Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "10.3.152.78:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 1433,
            "optime" : Timestamp(1563442691, 1),
            "optimeDate" : ISODate("2019-07-18T09:38:11Z"),
            "electionTime" : Timestamp(1563441462, 1),
            "electionDate" : ISODate("2019-07-18T09:17:42Z"),
            "configVersion" : 3,
            "self" : true
        },
        {
            "_id" : 2,
            "name" : "10.3.151.34:27020",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 1432,
            "lastHeartbeat" : ISODate("2019-07-18T09:41:31.679Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T09:41:32.973Z"),
            "pingMs" : 0,
            "configVersion" : 3
        },
        {
            "_id" : 3,
            "name" : "10.3.151.34:27018",           #成功
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 102,
            "optime" : Timestamp(1563442691, 1),
            "optimeDate" : ISODate("2019-07-18T09:38:11Z"),
            "lastHeartbeat" : ISODate("2019-07-18T09:41:31.690Z"),
            "lastHeartbeatRecv" : ISODate("2019-07-18T09:41:33.004Z"),
            "pingMs" : 0,
            "configVersion" : 3
        }
    ],
    "ok" : 1
}
testdb:PRIMARY> 

3. If you want to add one more node from

Operation method similar to the above method
1, a mounting mongodb

Guess you like

Origin blog.51cto.com/12965094/2421453