一个诡异的tcp连接异常问题

调试一tcp server程序遇到一个奇怪问题。在某单板上，client设备A只有开始2,3次可以正常connect()到server进行数据交互。再往后tcp connect就会超时。无论再怎么连都连不上。
诡异的是：
1. 当设备A无法正常tcp connect的时候。通过设备B或者其他pc机进行tcp连接server，可以正常连接。所以看起来server端是没什么问题的
2. 出问题的时候，重启server程序，重新建立socket，问题依旧，故障无法恢复

在server端通过tcpdump + wireshark抓包。发现server端收到了client发来的TCP SYN，但是没有回复SYN ACK。
任凭client重传若干次依旧不为所动。所以导致client端connect超时

下面是一个正常的连接，SYN后可以看到有恢复SYN ACK

netstat看不到任何异常现象。

通过查看nf conntrack。发现了灵异的事情。

root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786

ipv4 2 tcp 6 298 ESTABLISHED src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=7 bytes=1501 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 28 ESTABLISHED src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=8 bytes=1890 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 296 ESTABLISHED src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=10 bytes=2587 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 292 ESTABLISHED src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=10 bytes=2587 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 292 ESTABLISHED src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=11 bytes=2767 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 9 FIN_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=12 bytes=2819 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 8 FIN_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=12 bytes=2819 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 7 FIN_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=12 bytes=2819 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 6 FIN_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=12 bytes=2819 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 6 FIN_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=12 bytes=2819 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 29 CLOSE_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=13 bytes=2935 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 28 CLOSE_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=13 bytes=2935 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 27 CLOSE_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=13 bytes=2935 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 26 CLOSE_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=13 bytes=2935 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 2 CLOSE_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=13 bytes=2935 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 0 CLOSE_WAIT src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=4 bytes=624 src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=13 bytes=2935 [ASSURED] mark=327680 use=2
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 3604 ESTABLISHED src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=1 bytes=88 [UNREPLIED] src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=0 bytes=0 mark=327680 use=3
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 3602 ESTABLISHED src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=1 bytes=88 [UNREPLIED] src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=0 bytes=0 mark=327680 use=3
root@thw:~# grep 169 /proc/net/nf_conntrack | grep 786
ipv4 2 tcp 6 3600 ESTABLISHED src=169.254.29.1 dst=169.254.29.13 sport=786 dport=1024 packets=1 bytes=88 [UNREPLIED] src=169.254.29.13 dst=169.254.29.1 sport=1024 dport=786 packets=0 bytes=0 mark=327680 use=2

在一次正常连接后，tcp socket在CLOST_WAIT状态后收到了LAST ACK应该进入关闭状态，但是很灵异的在没有任何报文的情况下自己进入了established状态。这个状态下就不再响应任何TCP SYN报文了。而nf conntrack的默认timeout是3600秒

cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established
3600

只有设备A会出问题的原因也找到了。就是其他设备在做tcp连接的时候都会默认随机源端口。而只有设备A每次连接都会使用默认的1024端口。 DHCP缓存也会每次给设备A分配同样的IP地址。五元组不会变，导致命中异常的nf conntrack。无法正常建立tcp连接。

一个诡异的tcp连接异常问题

猜你喜欢