[Abnormal] Ali cloud ECS Redis build their own cluster, issue JedisCluster connection timeout occurs constantly

Others 2020-02-28 13:39:41 views: null

Description premise:

Service environment, using ECS Ali cloud cloud services. Three common services, with docker primary structures 3 3 6 clusters from the environment. Use springCloud do a distributed framework. Use lettuce to make a connection pool.

Reference mounting structures:
[SpringCloud distributed frame structures mounted redis5] docker-compose the cluster. And integrated into springCloud the

following error message:

org.springframework.dao.DataAccessResourceFailureException: Unable to connect to 172.16.75.43:6300; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to 172.16.75.43:6300
	at org.springframework.data.redis.connection.lettuce.LettuceClusterConnection$LettuceClusterNodeResourceProvider.getResourceForSpecificNode(LettuceClusterConnection.java:697) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.lettuce.LettuceClusterConnection$LettuceClusterNodeResourceProvider.getResourceForSpecificNode(LettuceClusterConnection.java:674) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnSingleNode(ClusterCommandExecutor.java:140) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnSingleNode(ClusterCommandExecutor.java:123) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnArbitraryNode(ClusterCommandExecutor.java:111) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.lettuce.LettuceClusterServerCommands.time(LettuceClusterServerCommands.java:317) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.DefaultedRedisConnection.time(DefaultedRedisConnection.java:1218) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
	at org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke(CloseSuppressingInvocationHandler.java:61) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at com.sun.proxy.$Proxy208.time(Unknown Source) ~[na:na]
	at com.keda.util.redis.RedisService$1.doInRedis(RedisService.java:840) ~[keda-util-1.1.2.jar!/:na]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:171) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at com.keda.util.redis.RedisService.currtTimeForRedis(RedisService.java:837) ~[keda-util-1.1.2.jar!/:na]
	at com.keda.util.redis.RedisService.getNowLocalDateTime(RedisService.java:873) ~[keda-util-1.1.2.jar!/:na]
	at com.keda.orderbuy.scheduled.timeTask.myTask(timeTask.java:50) ~[classes!/:0.0.1-SNAPSHOT]
	at com.keda.orderbuy.scheduled.timeTask$$FastClassBySpringCGLIB$$689dd3ff.invoke(<generated>) ~[classes!/:0.0.1-SNAPSHOT]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.cloud.sleuth.instrument.scheduling.TraceSchedulingAspect.traceBackgroundThread(TraceSchedulingAspect.java:73) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
	at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at com.keda.orderbuy.scheduled.timeTask$$EnhancerBySpringCGLIB$$27fb77d1.myTask(<generated>) ~[classes!/:0.0.1-SNAPSHOT]
	at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at net.javacrumbs.shedlock.core.DefaultLockingTaskExecutor.executeWithLock(DefaultLockingTaskExecutor.java:54) ~[shedlock-core-2.3.0.jar!/:na]
	at net.javacrumbs.shedlock.core.DefaultLockingTaskExecutor.executeWithLock(DefaultLockingTaskExecutor.java:39) ~[shedlock-core-2.3.0.jar!/:na]
	at net.javacrumbs.shedlock.core.DefaultLockManager.executeWithLock(DefaultLockManager.java:51) ~[shedlock-core-2.3.0.jar!/:na]
	at net.javacrumbs.shedlock.core.LockableRunnable.run(LockableRunnable.java:35) ~[shedlock-core-2.3.0.jar!/:na]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_232]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_232]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_232]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to 172.16.75.43:6300
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.getConnection(PooledClusterConnectionProvider.java:361) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.getConnection(PooledClusterConnectionProvider.java:346) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.StatefulRedisClusterConnectionImpl.getConnection(StatefulRedisClusterConnectionImpl.java:160) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at org.springframework.data.redis.connection.lettuce.LettuceClusterConnection$LettuceClusterNodeResourceProvider.getResourceForSpecificNode(LettuceClusterConnection.java:695) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	... 52 common frames omitted
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to 172.16.75.43:6300
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.lambda$getConnectionAsync$6(PooledClusterConnectionProvider.java:328) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_232]
	at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:631) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	... 1 common frames omitted
Caused by: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection timed out: /172.16.75.43:6300
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) ~[na:1.8.0_232]
	... 19 common frames omitted
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection timed out: /172.16.75.43:6300
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection timed out
	at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) ~[netty-transport-native-unix-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.channel.unix.Socket.finishConnect(Socket.java:243) ~[netty-transport-native-unix-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:667) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:644) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]

This problem has been tortured for several days. Redis.conf repeatedly check their profiles, and without any problems.
The point is, I RedisDesktopManager client, can connect all six nodes, and can set key, get key.

For this reason, I also wrote a 100 times for loop operations redis. Successfully get / set 85 times. Connection timeout, the number of errors caused by 15 times.

Moment on the bad people. Then all kinds of Baidu, are no solution. However, there is someone reminded the sentence:
" Ip address is not docker redis use, is not set to an address outside the network can access the "
paste my redis-cluster codes:

redis-cli -a KedaX3@1 \
--cluster create \
--cluster-replicas 1 \
101.xx.xx.160:6300 101.xx.xx.160:6301 \
47.xx.xx.148:6300 47.xx.xx.148:6301 \
47.xx.xx.44:6300   47.xx.xx.44:6301

Think, Ip configuration. We are outside the network can access. But the problem is, springCloud end time of the visit. How is connected docker ip address. Namely (172.16.75.43:6300).

I can only guess, when creating the cluster. To use the network address. So we are going to see, node configuration file. I.e., Nodes-6300.conf (file name, is disposed in the redis.conf. Follows :)

# 每个集群节点都有集群配置文件，这个文件不是用来提供给使用者编辑，
# 它将由集群节点自身创建并通过集群通信后修改。
# 每个集群节点如果在同一个宿主机上都要有不同的文件，否则将会被覆盖。
#
cluster-config-file nodes-6300.conf

nodes-6300.conf profile as follows:

c362f3db937xxxxxxxxxxxxxxxxxxxxxxxx3f2fe 47.xx.xx.148:6301@16301 slave 8fc4eabf8dcxxxxxxxxxxxxxxxxxxxxxxxxfe831 0 1576051320000 2 connected
8fc4eabf8dxxxxxxxxxxxxxxxxxxxxxxxxxfe831 172.16.75.43:6300@16300 master - 0 1576051320607 2 connected 5461-10922
67b016927xxxxxxxxxxxxxxxxxxxxxxxxxxx65f2 101.xx.xx.160:6301@16301 master - 0 1576051320000 5 connected 0-5460
fc485e421xxxxxxxxxxxxxxxxxxxxxxxxxfc9b 47.xx.xx.44:6301@16301 master - 0 1576051320662 4 connected 10923-16383
5eb87df6xxxxxxxxxxxxxxxxxxxxxxxxxf6e2f 101.xx.xx.160:6300@16300 slave fc485e4xxxxxxxxxxxxxxxxxxxxxxxxxx342fc9b 0 1576051320427 4 connected
d905ba8xxxxxxxxxxxxxxxxxxxxxxxxxxc69321d 47.xx.xx.148:6300@16300 myself,slave 67b0169xxxxxxxxxxxxxxxxxxxxxxxxxxc465f2 0 1576051320000 1 connected
vars currentEpoch 5 lastVoteEpoch 4

We can see at a glance, the second line is the network address, namely (172.16.75.43:6300@16300 master), not the external network accessible: 47.xx.xx.44: 6300

Modified to 47.xx.xx.44: 6300

Then turn to see the other five redis of node-xxxx.conf file. And all changes corresponding to the public network Ip. Restart redis docker. 


Problems the perfect solution! ! ! ! !

 

Old and new
Published 111 original articles · won praise 28 · views 40000 +
Private letter concerns

Guess you like

Origin blog.csdn.net/weixin_42697074/article/details/103574112
Recommended
Ranking
A quick primer on numpy basics
Two sister tease python project
Java | Multiple keywords in the replacement content are returned to the front end in red style
C ++: references
The string leetcood 1018 is a binary prefix divisible by 5
Flutter 布局组件
Java combat spingboot-redis
PMP pre-exam sprint and wrong questions sorting
ActiveMq C# Message Features: Delayed and Timed Message Delivery
DSP28377S_ copy a program from the RAM to FLASH portion DETAILS
Daily
More
2024-05-16(23)
2024-05-15(5)
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)
2024-05-09(32)
2024-05-08(18)
2024-05-07(34)

Code World

© 2018 codetd.com