引用
2016-07-14T16:42:10.779+0800 I NETWORK [LockPinger] Socket recv() timeout 10.0.0.6:30001
2016-07-14T16:42:10.779+0800 I NETWORK [LockPinger] SocketException: remote: 10.0.0.6:30001 error: 9001 socket exception [RECV_TIMEOUT] server [10.0.0.6:30001]
2016-07-14T16:42:10.779+0800 I NETWORK [LockPinger] DBClientCursor::init call() failed
2016-07-14T16:42:12.595+0800 I NETWORK [LockPinger] scoped connection to realsightback4:30001,realsightback3:30001,realsightback2:30001 not being returned to the pool
2016-07-14T16:42:12.595+0800 W SHARDING [LockPinger] distributed lock pinger'realsightback4:30001,realsightback3:30001,realsightback2:30001/realsightback1:30000:1468133855:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::update prepare failed: realsightback4:30001 (10.0.0.6) failed:10276 DBClientBase::findN: transport error: realsightback4:30001 ns: admin.$cmd query: { getlasterror: 1, fsync: 1 }
一开始看socket超时,以为是网络问题,用iperf排查没有问题,于是查看mongodb源码,感觉是config server的 { getlasterror: 1, fsync: 1 }命令超时,于是直接连接上config server执行该命令,三四十秒才返回,用iostat看,果然IO负载很高,于是多挂了一块盘,把不同shard的数据目录分开,问题解决了
原来aws上就只有一块盘,看来azure还是不如aws啊