本案中3个关键服务器
物理服务器：192.168.6.63，简称P，(Physical server)
KVM-VM：192.168.6.150，是物理服务器P上的一个KVM虚机，简称VM
NAS：外部NAS服务器，用来做ping/arp测试服务器，简称NAS
物理服务器P的配置：
#uname -a
Linux cz63 4.15.18-11-pve #1 SMP PVE 4.15.18-34 (Mon, 25 Feb 2019 14:51:06 +0100) x86_64 GNU/Linux
#cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto enp3s0f0
iface enp3s0f0 inet manual
auto enp3s0f1
iface enp3s0f1 inet manual
auto ens1f0
iface ens1f0 inet manual
auto ens1f1
iface ens1f1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp3s0f0 enp3s0f1
bond-miimon 100
bond-mode balance-rr
auto bond1
iface bond1 inet manual
bond-slaves ens1f0 ens1f1
bond-miimon 100
bond-mode balance-rr
auto vmbr0
iface vmbr0 inet static
address 192.168.6.63
netmask 255.255.255.0
gateway 192.168.6.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet static
address 10.1.1.63
netmask 255.255.255.0
bridge-ports bond1
bridge-stp off
bridge-fd 0
#brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.ac1f6b342094 no bond0
tap401000001i0
vmbr1 8000.74a4b500e768 no bond1
安装KVM-VM，之后VM与外部服务器的连接非常不稳定，90%情况下会出现ping不可达。这里测试了centos、ubuntu、win7都是一样的不稳定。
在VM内执行ping 192.168.6.40（NAS）
结果ping不通。
此时，在P上执行
tcpdump -leni vmbr0 arp
tcpdump -leni tap401000001i0 arp
发现情况如下：
P上
#tcpdump -leni vmbr0 arp | grep 2a:f0:5f:ae:c9:8b
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
P上
#tcpdump -leni tap401000001i0 arp | grep 2a:f0:5f:ae:c9:8b
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
NAS上
# tcpdump -leni vmx0 arp | grep c9:8b
02:03:50.930907 2a:f0:5f:ae:c9:8b > 00:50:56:87:86:b9, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 tell 192.168.6.150, length 46
02:03:50.930923 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 42: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 28
02:04:02.669823 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:02.670131 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:03.670770 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:03.671059 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:04.672736 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:04.672992 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:06.671878 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:06.672021 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:07.674726 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:07.674773 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:08.676733 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:08.676868 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:10.673678 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:10.674026 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
vmbr0上发现了2个request包和1个reply包
tap401000001i0上仅发现了request包，而且length不一致，未发现reply包。
NAS上一切正常
看到的现象就是VM在发送icmp之前需要先获得NAS的MAC地址，所以需要先通过ARP广播获取到NAS的MAC地址。
- P的vmbr0异常：接收到2个request，1个reply
- P的vmbr0异常：接收到reply未发送至tap401000001i0
由于以上2个原因，导致VM无法收到reply，所以icmp不能进行，ping失败。
问题定位在数据链路层的ARP协议上。
在VM上执行arping -c 10 192.168.6.40
现象依据。再次证明以上2个现象。
# brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.ac1f6b342094 no bond0
tap401000001i0
vmbr1 8000.74a4b500e768 no bond1
# brctl showstp vmbr0
vmbr0
bridge id 8000.ac1f6b342094
designated root 8000.ac1f6b342094
root port 0 path cost 0
max age 20.00 bridge max age 20.00
hello time 2.00 bridge hello time 2.00
forward delay 0.00 bridge forward delay 0.00
ageing time 300.00
hello timer 0.00 tcn timer 0.00
topology change timer 0.00 gc timer 4.18
flags
bond0 (1)
port id 8001 state forwarding
designated root 8000.ac1f6b342094 path cost 4
designated bridge 8000.ac1f6b342094 message age timer 0.00
designated port 8001 forward delay timer 0.00
designated cost 0 hold timer 0.00
flags
tap401000001i0 (2)
port id 8002 state forwarding
designated root 8000.ac1f6b342094 path cost 100
designated bridge 8000.ac1f6b342094 message age timer 0.00
designated port 8002 forward delay timer 0.00
designated cost 0 hold timer 0.00
flags
# brctl showmacs vmbr0
port no mac addr is local? ageing timer
1 2a:f0:5f:ae:c9:8b no 1.13
1 00:50:56:87:e3:6d no 0.00
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# ebtables -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
# ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
# ip route show table all
default via 192.168.6.1 dev vmbr0 onlink
10.1.1.0/24 dev vmbr1 proto kernel scope link src 10.1.1.63
192.168.6.0/24 dev vmbr0 proto kernel scope link src 192.168.6.63
broadcast 10.1.1.0 dev vmbr1 table local proto kernel scope link src 10.1.1.63
local 10.1.1.63 dev vmbr1 table local proto kernel scope host src 10.1.1.63
broadcast 10.1.1.255 dev vmbr1 table local proto kernel scope link src 10.1.1.63
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.6.0 dev vmbr0 table local proto kernel scope link src 192.168.6.63
local 192.168.6.63 dev vmbr0 table local proto kernel scope host src 192.168.6.63
broadcast 192.168.6.255 dev vmbr0 table local proto kernel scope link src 192.168.6.63
# bridge vlan show
port vlan ids
bond0 1 PVID Egress Untagged
vmbr0 1 PVID Egress Untagged
bond1 1 PVID Egress Untagged
vmbr1 1 PVID Egress Untagged
tap401000001i0 1 PVID Egress Untagged以上一切正常。
sequenceDiagram
A->>B: How are you?
B->>A: Great!