【异常】阿里云ECS自己搭建Redis集群，不停的出现JedisCluster连接超时的问题

其他 2020-02-28 13:07:31 阅读次数: 0

前提说明：

服务环境，用的是阿里云的 ECS云服务。 共用三台服务，用docker 搭建 3 主3 从共6个集群环境。 使用springCloud做分布式框架。 使用 lettuce 做连接池。

安装搭建参考:
【SpringCloud分布式框架搭建】docker-compose安装redis5 集群。并集成到springCloud中

错误信息如下：

org.springframework.dao.DataAccessResourceFailureException: Unable to connect to 172.16.75.43:6300; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to 172.16.75.43:6300
	at org.springframework.data.redis.connection.lettuce.LettuceClusterConnection$LettuceClusterNodeResourceProvider.getResourceForSpecificNode(LettuceClusterConnection.java:697) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.lettuce.LettuceClusterConnection$LettuceClusterNodeResourceProvider.getResourceForSpecificNode(LettuceClusterConnection.java:674) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnSingleNode(ClusterCommandExecutor.java:140) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnSingleNode(ClusterCommandExecutor.java:123) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnArbitraryNode(ClusterCommandExecutor.java:111) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.lettuce.LettuceClusterServerCommands.time(LettuceClusterServerCommands.java:317) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.connection.DefaultedRedisConnection.time(DefaultedRedisConnection.java:1218) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
	at org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke(CloseSuppressingInvocationHandler.java:61) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at com.sun.proxy.$Proxy208.time(Unknown Source) ~[na:na]
	at com.keda.util.redis.RedisService$1.doInRedis(RedisService.java:840) ~[keda-util-1.1.2.jar!/:na]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:171) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	at com.keda.util.redis.RedisService.currtTimeForRedis(RedisService.java:837) ~[keda-util-1.1.2.jar!/:na]
	at com.keda.util.redis.RedisService.getNowLocalDateTime(RedisService.java:873) ~[keda-util-1.1.2.jar!/:na]
	at com.keda.orderbuy.scheduled.timeTask.myTask(timeTask.java:50) ~[classes!/:0.0.1-SNAPSHOT]
	at com.keda.orderbuy.scheduled.timeTask$$FastClassBySpringCGLIB$$689dd3ff.invoke(<generated>) ~[classes!/:0.0.1-SNAPSHOT]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.cloud.sleuth.instrument.scheduling.TraceSchedulingAspect.traceBackgroundThread(TraceSchedulingAspect.java:73) ~[spring-cloud-sleuth-core-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
	at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at com.keda.orderbuy.scheduled.timeTask$$EnhancerBySpringCGLIB$$27fb77d1.myTask(<generated>) ~[classes!/:0.0.1-SNAPSHOT]
	at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at net.javacrumbs.shedlock.core.DefaultLockingTaskExecutor.executeWithLock(DefaultLockingTaskExecutor.java:54) ~[shedlock-core-2.3.0.jar!/:na]
	at net.javacrumbs.shedlock.core.DefaultLockingTaskExecutor.executeWithLock(DefaultLockingTaskExecutor.java:39) ~[shedlock-core-2.3.0.jar!/:na]
	at net.javacrumbs.shedlock.core.DefaultLockManager.executeWithLock(DefaultLockManager.java:51) ~[shedlock-core-2.3.0.jar!/:na]
	at net.javacrumbs.shedlock.core.LockableRunnable.run(LockableRunnable.java:35) ~[shedlock-core-2.3.0.jar!/:na]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_232]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_232]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_232]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to 172.16.75.43:6300
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.getConnection(PooledClusterConnectionProvider.java:361) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.getConnection(PooledClusterConnectionProvider.java:346) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.StatefulRedisClusterConnectionImpl.getConnection(StatefulRedisClusterConnectionImpl.java:160) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at org.springframework.data.redis.connection.lettuce.LettuceClusterConnection$LettuceClusterNodeResourceProvider.getResourceForSpecificNode(LettuceClusterConnection.java:695) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
	... 52 common frames omitted
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to 172.16.75.43:6300
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.lambda$getConnectionAsync$6(PooledClusterConnectionProvider.java:328) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_232]
	at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:631) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	... 1 common frames omitted
Caused by: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection timed out: /172.16.75.43:6300
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607) ~[na:1.8.0_232]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) ~[na:1.8.0_232]
	... 19 common frames omitted
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection timed out: /172.16.75.43:6300
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection timed out
	at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) ~[netty-transport-native-unix-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.channel.unix.Socket.finishConnect(Socket.java:243) ~[netty-transport-native-unix-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:667) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:644) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) ~[netty-transport-native-epoll-4.1.39.Final-linux-x86_64.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.39.Final.jar!/:4.1.39.Final]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]

一直被这问题，折磨了几天。 反复检查自己的 redis.conf 配置文件，并没有任何问题。
重点是，我用 RedisDesktopManager客户端，可以连接 所有的 6个节点，并且能set key,get key .

为此，我还写了一个 100次的for循环，操作redis。 成功get/set了85次。 连接超时，导致的错误次数为 15次。

瞬间人就不好了。然后各种百度，都无解。 但是，有个网友提醒了一句：
“是不是docker redis 使用的Ip地址,没有设置成外网可访问的地址”
贴上我的 redis-cluster代码：

redis-cli -a KedaX3@1 \
--cluster create \
--cluster-replicas 1 \
101.xx.xx.160:6300 101.xx.xx.160:6301 \
47.xx.xx.148:6300 47.xx.xx.148:6301 \
47.xx.xx.44:6300   47.xx.xx.44:6301

自认为，配置的Ip。 都是外网能访问的。 但是问题是，springCloud端，访问的时候。 连接的怎么会是 docker 的 ip地址。即（172.16.75.43:6300）。

只能猜，在创建集群的时候。 使用到了 内网地址。  所以打算去看，节点配置文件。即 nodes-6300.conf（文件名，是在 redis.conf 中配置的。如下：）

# 每个集群节点都有集群配置文件，这个文件不是用来提供给使用者编辑，
# 它将由集群节点自身创建并通过集群通信后修改。
# 每个集群节点如果在同一个宿主机上都要有不同的文件，否则将会被覆盖。
#
cluster-config-file nodes-6300.conf

nodes-6300.conf 配置文件内容如下：

c362f3db937xxxxxxxxxxxxxxxxxxxxxxxx3f2fe 47.xx.xx.148:6301@16301 slave 8fc4eabf8dcxxxxxxxxxxxxxxxxxxxxxxxxfe831 0 1576051320000 2 connected
8fc4eabf8dxxxxxxxxxxxxxxxxxxxxxxxxxfe831 172.16.75.43:6300@16300 master - 0 1576051320607 2 connected 5461-10922
67b016927xxxxxxxxxxxxxxxxxxxxxxxxxxx65f2 101.xx.xx.160:6301@16301 master - 0 1576051320000 5 connected 0-5460
fc485e421xxxxxxxxxxxxxxxxxxxxxxxxxfc9b 47.xx.xx.44:6301@16301 master - 0 1576051320662 4 connected 10923-16383
5eb87df6xxxxxxxxxxxxxxxxxxxxxxxxxf6e2f 101.xx.xx.160:6300@16300 slave fc485e4xxxxxxxxxxxxxxxxxxxxxxxxxx342fc9b 0 1576051320427 4 connected
d905ba8xxxxxxxxxxxxxxxxxxxxxxxxxxc69321d 47.xx.xx.148:6300@16300 myself,slave 67b0169xxxxxxxxxxxxxxxxxxxxxxxxxxc465f2 0 1576051320000 1 connected
vars currentEpoch 5 lastVoteEpoch 4

可以一眼就看到，第二行就是 内网地址，即（172.16.75.43:6300@16300 master），并不是 外网可访问的：47.xx.xx.44:6300

修改 成 47.xx.xx.44:6300

然后依次，查看其他5台 redis的node-xxxx.conf文件。 并全部修改成对应的 公网Ip.重启 redis docker. 


问题完美解决了！！！！！

老新人
发布了111 篇原创文章 · 获赞 28 · 访问量 4万+
私信 关注

猜你喜欢

转载自blog.csdn.net/weixin_42697074/article/details/103574112
今日推荐
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
周排行
NEFU 117 素数个数的位数
Closest Common Ancestors (Lca,tarjan)
ELK部署
【转载】Hive笔记整理（三）
SQL语句（一）基本表的定义
关于Java web开发中的MySQL的事务语句
MFC创建自定义窗体
如何用一句话激怒程序员？
《逆袭大学》文摘——9.4 基础和应用的平衡中找到大学的节奏
【spring源码分析】@Value注解原理
每日归档
更多
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022