2017-12-20 03:02:57.044 WARN (main-SendThread(10.76.17.14:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.14/10.76.17.14:2181, unexpected error, closing socket co
nnection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 718 with err -101 expected Xid 717 for a packet with details: clientPath:null serverPath:null finished:false header:: 717,14 replyHeader:: 0,0,
-4 request:: org.apache.zookeeper.MultiTransactionRecord@dd1ab83c response:: org.apache.zookeeper.MultiResponse@0
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-12-20 03:02:57.049 ERROR (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext There was a problem trying
to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:218)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:417)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:308)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1091)
at org.apache.solr.cloud.ZkController.register(ZkController.java:894)
at org.apache.solr.cloud.ZkController.register(ZkController.java:849)
at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:212)
at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:697)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:821)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:751)
at org.apache.solr.handler.admin.CoreAdminOperation$1.call(CoreAdminOperation.java:131)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:153)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:935)
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:570)
at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:567)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.multi(SolrZkClient.java:567)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:201)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:49)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:42)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:183)
... 40 more 2017-12-20 03:03:25.675 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext There may be a better lead
er candidate than us - going back into recovery
2017-12-20 03:03:25.675 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ElectionContext Canceling election /collections/hospi
tal_resources/leader_elect/shard1/election/171210747425456190-core_node1-n_0000000002
2017-12-20 03:03:25.774 ERROR (OverseerCollectionConfigSetProcessor-171210747425456190-10.76.17.14:8983_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:358)
at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:175)
at java.lang.Thread.run(Thread.java:745)
2017-12-20 03:03:25.776 INFO (zkCallback-4-thread-43-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam
e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2017-12-20 03:03:25.776 INFO (zkCallback-4-thread-43-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2017-12-20 03:03:26.018 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2017-12-20 03:03:26.020 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 name
:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2017-12-20 03:03:27.327 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContextBase No version found for e
phemeral leader parent node, won't remove previous leader registration.
2017-12-20 03:03:27.329 WARN (updateExecutor-2-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_resour
ces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[hospital_resources_shard1_replica1] coreNodeName=[core_node1]
2017-12-20 03:03:27.329 INFO (updateExecutor-2-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_resour
ces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.u.DefaultSolrCoreState Running recovery
2017-12-20 03:03:27.329 INFO (updateExecutor-2-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_resour
ces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ActionThrottle The last recovery attempt started 28638ms ago.
2017-12-20 03:03:27.331 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso
urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
2017-12-20 03:03:27.331 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso
urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
2017-12-20 03:03:27.342 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.LeaderElector Joined leadership election with path: /
collections/hospital_resources/leader_elect/shard1/election/171210747425456190-core_node1-n_0000000004
2017-12-20 03:03:27.344 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ActionThrottle The last leader attempt started 28639m
s ago.
2017-12-20 03:03:27.345 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Running the leader process
for shard=shard1 and weAreReplacement=true and leaderVoteWait=180000
2017-12-20 03:03:27.381 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext All replicas are ready to
participate in election.
2017-12-20 03:03:27.382 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Checking if I should try a
nd be the leader.
2017-12-20 03:03:27.382 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext My last published State wa
s Active, it's okay to be the leader.
2017-12-20 03:03:27.382 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader -
try and sync
2017-12-20 03:03:27.382 WARN (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[hospital
_resources_shard1_replica1] coreNodeName=[core_node1]
2017-12-20 03:03:27.383 INFO (OverseerStateUpdate-171210747425456190-10.76.17.14:8983_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
"operation":"leader",
"shard":"shard1",
"collection":"hospital_resources"} current state version: 0
2017-12-20 03:03:27.406 INFO (OverseerStateUpdate-171210747425456190-10.76.17.14:8983_solr-n_0000000000) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/hospital_resour
ces/state.json version: 3
2017-12-20 03:03:27.439 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataC
hanged path:/collections/hospital_resources/state.json] for collection [hospital_resources] has occurred - updating... (live nodes size: [1])
2017-12-20 03:03:27.442 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updating data for [hospital_resources] from [3] to [4]
2017-12-20 03:03:29.882 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://10.76.17.14:8983
/solr/hospital_resources_shard1_replica1/
2017-12-20 03:03:29.883 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2017-12-20 03:03:29.884 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.SyncStrategy http://10.76.17.14:8983/solr/hospital_re
sources_shard1_replica1/ has no replicas
2017-12-20 03:03:29.889 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContextBase Creating leader regist
ration node /collections/hospital_resources/leaders/shard1/leader after winning as /collections/hospital_resources/leader_elect/shard1/election/171210747425456190-core_node1-n_0000000004
2017-12-20 03:03:30.031 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.12/10.76.17.12:2181, unexpected error, closing socket co
nnection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 804 with err 0 expected Xid 803 for a packet with details: clientPath:null serverPath:null finished:false header:: 803,14 replyHeader:: 0,0,-4
request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-12-20 03:03:30.134 INFO (zkCallback-4-thread-31-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam
e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2017-12-20 03:03:30.134 INFO (zkCallback-4-thread-31-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2017-12-20 03:03:31.065 WARN (main-SendThread(10.76.17.11:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2017-12-20 03:03:31.067 INFO (zkCallback-4-thread-7-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 name
:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2017-12-20 03:03:31.353 ERROR (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso
urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Error while trying to recover. core=hospital_resources_shard1_replica1:org.apache.solr.common.SolrEx
ception: No registered leader was found after waiting for 4000ms , collection: hospital_resources slice: shard1
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:626)
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:612)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:306)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-12-20 03:03:31.353 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso
urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates.
2017-12-20 03:03:31.353 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso
urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed
2017-12-20 03:03:31.353 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso
urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
2017-12-20 03:03:31.636 WARN (main-SendThread(10.76.17.11:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.11/10.76.17.11:2181, unexpected error, closing socket co
nnection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 807 with err 0 expected Xid 806 for a packet with details: clientPath:null serverPath:null finished:false header:: 806,14 replyHeader:: 0,0,-4
request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-12-20 03:03:31.739 INFO (zkCallback-4-thread-33-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam
e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2017-12-20 03:03:31.740 INFO (zkCallback-4-thread-33-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2017-12-20 03:03:32.487 WARN (main-SendThread(10.76.17.14:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2017-12-20 03:03:32.490 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 name
:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2017-12-20 03:03:34.741 WARN (main-SendThread(10.76.17.14:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.14/10.76.17.14:2181, unexpected error, closing socket co
nnection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 810 with err 0 expected Xid 809 for a packet with details: clientPath:null serverPath:null finished:false header:: 809,14 replyHeader:: 0,0,-4
request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-12-20 03:03:34.844 INFO (zkCallback-4-thread-32-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam
e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
2017-12-20 03:03:34.845 INFO (zkCallback-4-thread-32-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2017-12-20 03:03:35.526 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2017-12-20 03:03:35.529 INFO (zkCallback-4-thread-43-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam
e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2017-12-20 03:03:39.346 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.12/10.76.17.12:2181, unexpected error, closing socket co
nnection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 814 with err 0 expected Xid 813 for a packet with details: clientPath:null serverPath:null finished:false header:: 813,14 replyHeader:: 0,0,-4
request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
|
