日常工作问题解决：rhel7下使用teamd配置双网卡绑定

1、情景描述

因为业务发展，需要部署一套ORACLE 11g RAC实现高可用，所以公司新分配两台物理机，每台物理机配置4张网卡，2张千兆，2张万兆

现规划如下：

2张千兆网卡，做双网卡绑定使用主备模式，作为心跳
2张万兆网卡，做双网卡绑定使用负载均衡模式，作为业务

使用rhel7下的NetworkManager服务组中的team服务来实现双网卡绑定链路聚合。

2、准备工作

2.1 确认网卡信息

使用ifconfig命令查看确认网卡信息

[root@db1 ~]$ ifconfig 

enp2s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 32:6e:d8:f5:d7:32  txqueuelen 1000  (Ethernet)
        RX packets 5924926  bytes 5114985550 (4.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4857883  bytes 3559876055 (3.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x96d00000-96dfffff  

enp2s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 32:6e:d8:f5:d7:32  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x96c00000-96cfffff  

enp5s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 46:be:3b:ac:2c:2e  txqueuelen 1000  (Ethernet)
        RX packets 1098  bytes 120567 (117.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp5s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 46:be:3b:ac:2c:2e  txqueuelen 1000  (Ethernet)
        RX packets 49209  bytes 17138608 (16.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55533  bytes 20284230 (19.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 606440  bytes 534199950 (509.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 606440  bytes 534199950 (509.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

发现4张网卡enp2s0f0、enp2s0f01、enp5s0f0、enp5s0f1

使用ethtool确认网卡信息：

[root@db1 ~]$ ethtool enp2s0f0
Settings for enp2s0f0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s  # 速度为1000mb/s，确认为千兆网卡
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: off (auto)
Cannot get wake-on-lan settings: Operation not permitted
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

[root@db1 ~]$ ethtool enp5s0f0
Settings for enp5s0f0:
    Supported ports: [ FIBRE ]
    Supported link modes:   10000baseT/Full 
    Supported pause frame use: Symmetric
    Supports auto-negotiation: No
    Advertised link modes:  10000baseT/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Speed: 10000Mb/s # 速度为10000mb/s，确认为万兆网卡
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

2.2 删除原有网卡配置信息

[root@db1 ~]$ nmcli connection show
名称                UUID                                  类型            设备         
beatteam0           0445cabf-c120-453a-a747-2aca7e8618a0  team            beatteam0    
beatteam0-port0     851acd5f-8b5b-46be-a507-a70319074085  802-3-ethernet  enp2s0f0     
beatteam0-port1     775ff22d-cee7-4f3b-add3-dc4ad56f5c6f  802-3-ethernet  enp2s0f1     
serviceteam0        00f01df1-0185-4e31-bd6d-42e4e18d771f  team            serviceteam0 
serviceteam0-port0  65d46a63-4c94-4482-b2ee-39d39e0cbdb4  802-3-ethernet  enp5s0f0     
serviceteam0-port0  41a55302-5c94-4a17-a0fc-d6b9c12f4624  802-3-ethernet  enp5s0f1     
virbr0              1c4011cb-4d0a-4880-9a4a-05284536fbcb  bridge          virbr0  
[root@db1 ~]$ nmcli connection delete 41a55302-5c94-4a17-a0fc-d6b9c12f4624 
# nmcli connection delete 设备UUID 删除UUID对应的网卡配置

确认所有将要使用的四张网卡信息已经全部删除以后，准备工作完成

3、配置网卡绑定

3.1 配置千兆网卡双网卡热备用作心跳

[root@db1 ~]$nmcli connection add type team con-name beatteam0 ifname beatteam0 config '{"runner":{"name":"activebackup"}}'
# 创建双网卡聚合team，team名为beatteam0，虚拟网卡名为beatteam0，方式为activebackup--热备
[root@db1 ~]$nmcli connection modify beatteam0 ipv4.addresses 192.168.10.21/24 # 配置IPV4地址和掩码
[root@db1 ~]$nmcli connection modify beatteam0 ipv4.gateway “192.168.10.1” # 配置IPV4网关
[root@db1 ~]$nmcli connection modify beatteam0 ipv4.dns '192.168.10.21 8.8.8.8' # 配置IPV4DNS
[root@db1 ~]$nmcli connection modify beatteam0 ipv4.method manual # 配置IPV4协议
[root@db1 ~]$nmcli connection add type team-slave con-name beatteam0-port0 ifname enp2s0f0 master beatteam0  
# 将enp2s0f0加入beatteam0，命名为beatteam0-port0
[root@db1 ~]$nmcli connection add type team-slave con-name beatteam0-port1 ifname enp2s0f1 master beatteam0
# 将enp2s0f1加入beatteam0，命名为beatteam0-port1
[root@db1 ~]$systemctl restart network # 重启网络服务
[root@db1 ~]$ nmcli connection show # 查看网络连接信息，发现已经添加成功
名称                UUID                                  类型            设备         
beatteam0           0445cabf-c120-453a-a747-2aca7e8618a0  team            beatteam0    
beatteam0-port0     851acd5f-8b5b-46be-a507-a70319074085  802-3-ethernet  enp2s0f0     
beatteam0-port1     775ff22d-cee7-4f3b-add3-dc4ad56f5c6f  802-3-ethernet  enp2s0f1  
[root@db1 ~]# teamdctl beatteam0 state # 查看team状态，发现争产
setup:
  runner: activebackup # 热备状态
ports:
  enp2s0f0
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up # 启动状态
        down count: 0
  enp2s0f1
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up # 启动状态
        down count: 0
runner:
  active port: enp2s0f0 # enp2s0f0为激活网卡

3.2 配置网兆网卡双网卡负载均衡用作业务

[root@db1 ~]$nmcli connection add type team con-name serviceteam0 ifname serviceteam0 config '{"runner":{"name":"loadbalance"}}'
# 创建双网卡聚合team，team名为serviceteam0，虚拟网卡名为serviceteam0，方式为activebackup--热备
[root@db1 ~]$nmcli connection modify serviceteam0 ipv4.addresses 192.168.20.51/24 # 配置IPV4地址和掩码
[root@db1 ~]$nmcli connection modify serviceteam0 ipv4.gateway “192.168.20.1” # 配置IPV4网关
[root@db1 ~]$nmcli connection modify serviceteam0 ipv4.dns '192.168.20.51 8.8.8.8' # 配置IPV4DNS
[root@db1 ~]$nmcli connection modify serviceteam0 ipv4.method manual # 配置IPV4协议
[root@db1 ~]$nmcli connection add type team-slave con-name serviceteam0-port0 ifname enp5s0f0 master serviceteam0  
# 将enp5s0f0加入serviceteam0，命名为serviceteam0-port0
[root@db1 ~]$nmcli connection add type team-slave con-name serviceteam0-port1 ifname enp5s0f1 master serviceteam0
# 将enp5s0f1加入serviceteam0，命名为serviceteam0-port1
[root@db1 ~]$systemctl restart network # 重启网络服务
[root@db1 ~]$ nmcli connection show # 查看网络连接，发现已经添加成功
名称                UUID                                  类型            设备         
beatteam0           0445cabf-c120-453a-a747-2aca7e8618a0  team            beatteam0    
beatteam0-port0     851acd5f-8b5b-46be-a507-a70319074085  802-3-ethernet  enp2s0f0     
beatteam0-port1     775ff22d-cee7-4f3b-add3-dc4ad56f5c6f  802-3-ethernet  enp2s0f1     
serviceteam0        00f01df1-0185-4e31-bd6d-42e4e18d771f  team            serviceteam0 
serviceteam0-port0  65d46a63-4c94-4482-b2ee-39d39e0cbdb4  802-3-ethernet  enp5s0f0     
serviceteam0-port0  41a55302-5c94-4a17-a0fc-d6b9c12f4624  802-3-ethernet  enp5s0f1     
virbr0              1c4011cb-4d0a-4880-9a4a-05284536fbcb  bridge          virbr0   
[root@db1 ~]# teamdctl serviceteam0 state # 查看serviceteam0的team状态，
setup:
  runner: loadbalance # 运行模式为loadbalance，负载均衡
ports:
  enp5s0f0
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up # 连接启动，状态正常
        down count: 0
  enp5s0f1
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up # 连接启动，状态正常
        down count: 0

按此办法配置另一台主机，配置完成。