Tungsten Fabric入门宝典丨关于集群更新的那些事

企业开发 2020-05-22 17:04:17 阅读次数: 0

Tungsten Fabric入门宝典系列文章，来自技术大牛倾囊相授的实践经验，由TF中文社区为您编译呈现，旨在帮助新手深入理解TF的运行、安装、集成、调试等全流程。如果您有相关经验或疑问，欢迎与我们互动，并与社区极客们进一步交流。更多TF技术文章，请点击公号底部按钮＞学习＞文章合集。

作者：Tatsuya Naganawa 译者：TF编译组

集群范围内的更新是很重要的功能，可以在保证生产集群的SLA的条件下提供集群中最新功能的有效性。

由于Tungsten Fabric在MPLS-***中使用类似的协议，因此，根据我的尝试，即使Control的模块版本和vRouter的模块版本不同，基本的互操作性还是有的。

因此，总体思路是，首先一个一个地更新controller，然后并根据需要使用vMotion或maintence模式逐一更新vRouter。

另外，Tungsten Fabric controller还支持名为ISSU的奇特功能，但是我认为这个名称有些混乱，因为Tungsten Fabric controller与route reflector非常相似，而不是routing-engine。

因此，基本思路是，首先将所有配置（configs）复制到新创建的controller（或route reflectors），然后更新vRouter设置（如果服务器可以重新启动，则更新vRouter模块）以使用这些新的controller。通过这个过程，vRouter模块更新的回滚操作也将更加容易。

下面我来描述一下这个过程。

就地更新（in-place update）

由于ansible-deployer遵循等幂行为（idempotent behavior），因此更新与安装没有太大区别。以下的命令将更新所有模块。

cd contrail-ansible-deployer
git pull
vi config/instances.yaml
(update CONTRAIL_CONTAINER_TAG)
ansible-playbook -e orchestrator=xxx -i inventory/ playbooks/install_contrail.yml

一个限制是由于此命令几乎同时重启了所有节点，因此不容易做到一一重启controller和vRouter。另外，从instances.yaml中删除其它节点将不起作用，因为一个节点的更新需要其它节点的某些参数。

  • 例如，vRouter更新需要控件的IP，该IP是从instances.yaml中的control role节点推导出的

为了克服这个问题，从R2005版本开始，ziu.yaml就加入了这个功能，至少对于控制平面来说，要逐一更新。

cd contrail-ansible-deployer
git pull
vi config/instances.yaml
(update CONTRAIL_CONTAINER_TAG)
ansible-playbook -e orchestrator=xxx -i inventory/ playbooks/ziu.yml
ansible-playbook -e orchestrator=xxx -i inventory/ playbooks/install_contrail.yml

就我试过的情况来看，当控制平面更新时，它会进行串联更新和重启控制进程，所以没有看到掉包的现象。

  • 在install_contrail.yaml期间，控制进程的重启是被跳过的，因为它们已经更新了。

  • 当执行vrouter-agent重启时，仍然会出现一些丢包现象，所以如果可以的话，建议进行工作负载迁移。

ISSU

即使容器格式的差异很大（例如从4.x到5.x），我们也可以使用ISSU，因为它创建了一个新的controller集群，并在其中复制数据。

首先，我来描述最简单的情况，即一个旧controller和一个新controller，以查看整个过程。所有命令都在新的controller上键入。

old-controller:
 ip: 172.31.2.209
 hostname: ip-172-31-2-209
new-controller:
 ip: 172.31.1.154
 hostname: ip-172-31-1-154

(both controllers are installed with this instances.yaml)
provider_config:
  bms:
   ssh_user: root
   ssh_public_key: /root/.ssh/id_rsa.pub
   ssh_private_key: /root/.ssh/id_rsa
   domainsuffix: local
   ntpserver: 0.centos.pool.ntp.org
instances:
  bms1:
   provider: bms
   roles:
      config_database:
      config:
      control:
      analytics:
      analytics_database:
      webui:
   ip: x.x.x.x ## controller's ip
contrail_configuration:
  CONTRAIL_CONTAINER_TAG: r5.1
  KUBERNETES_CLUSTER_PROJECT: {}
  JVM_EXTRA_OPTS: "-Xms128m -Xmx1g"
global_configuration:
  CONTAINER_REGISTRY: tungstenfabric

[commands]
1. 停止批处理作业
docker stop config_devicemgr_1
docker stop config_schema_1
docker stop config_svcmonitor_1

2. 在cassandra上注册新的control，在它们之间运行bgp
docker exec -it config_api_1 bash
python /opt/contrail/utils/provision_control.py --host_name ip-172-31-1-154.local --host_ip 172.31.1.154 --api_server_ip 172.31.2.209 --api_server_port 8082 --oper add --router_asn 64512 --ibgp_auto_mesh

3. 在控制器之间同步数据
vi contrail-issu.conf
(write down this)
[DEFAULTS]
old_rabbit_address_list = 172.31.2.209
old_rabbit_port = 5673
new_rabbit_address_list = 172.31.1.154
new_rabbit_port = 5673
old_cassandra_address_list = 172.31.2.209:9161
old_zookeeper_address_list = 172.31.2.209:2181
new_cassandra_address_list = 172.31.1.154:9161
new_zookeeper_address_list = 172.31.1.154:2181
new_api_info={"172.31.1.154": [("root"), ("password")]} ## ssh public-key can be used

image_id=`docker images | awk '/config-api/{print $3}' | head -1`

docker run --rm -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh $image_id -c "/usr/bin/contrail-issu-pre-sync -c /etc/contrail/contrail-issu.conf"

4. 启动进程进行实时数据同步
docker run --rm --detach -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh --name issu-run-sync $image_id -c "/usr/bin/contrail-issu-run-sync -c /etc/contrail/contrail-issu.conf"

(如果需要，请检查日志)
docker exec -t issu-run-sync tail -f /var/log/contrail/issu_contrail_run_sync.log

5. (更新 vrouters)

6. 在结束后停止作业，同步所有数据
docker rm -f issu-run-sync

image_id=`docker images | awk '/config-api/{print $3}' | head -1`
docker run --rm -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh --name issu-run-sync $image_id -c "/usr/bin/contrail-issu-post-sync -c /etc/contrail/contrail-issu.conf"
docker run --rm -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh --name issu-run-sync $image_id -c "/usr/bin/contrail-issu-zk-sync -c /etc/contrail/contrail-issu.conf"

7. 从cassandra中去掉旧的节点，加入新的节点
vi issu.conf
(write down this)
[DEFAULTS]
db_host_info={"172.31.1.154": "ip-172-31-1-154.local"}
config_host_info={"172.31.1.154": "ip-172-31-1-154.local"}
analytics_host_info={"172.31.1.154": "ip-172-31-1-154.local"}
control_host_info={"172.31.1.154": "ip-172-31-1-154.local"}
api_server_ip=172.31.1.154

docker cp issu.conf config_api_1:issu.conf
docker exec -it config_api_1 python /opt/contrail/utils/provision_issu.py -c issu.conf

8. 启动批处理作业
docker start config_devicemgr_1
docker start config_schema_1
docker start config_svcmonitor_1

以下将是可能的检查点。

  1. 步骤3之后，你可以尝试使用contrail-api-cli ls -l *来查看所有数据是否已成功复制，并且可以通过ist.py ctr nei来查看controller之间的ibgp是否已启动。

  2. 步骤4之后，你可以修改旧数据库，以查看更改是否可以成功传播到新数据库。

接下来，我将在使用编排器和两个vRouter的条件下，讨论更为实际的情况。

编排器集成

为了说明结合编排器的情况，我尝试用ansible-deployer部署两个vRouters和kubernetes。

即使与编排器结合使用，总体过程也不会有太大不同。

需要注意的是，需要在什么时候将kube-manager更改为新的。

从某种意义上讲，由于kube-manager动态订阅了来自kube-apiserver的事件并更新了Tungsten Fabric配置数据库（config-database），因此它类似于批处理作业，例如schema-transformer、svc-monitor和device-manager。因此，我使用此类批处理作业，同时停止并启动了旧的或新的kube-manager（实际上也包括webui），但是可能需要根据每个设置进行更改。

这个示例中的总体过程如下所示。

1.设置一个controller（带有一个kube-manager和kubernetes-master）和两个vRouter
2.设置一个新的controller（带有一个kube-manager，但kubernetes-master与旧的controller相同）
3.停止批处理作业、新controller的kube-manager和webui
4.启动ISSU进程并继续执行，直到开始运行同步（run-sync）
  ->将在controller之间建立iBGP
5.根据新controller的ansible-ployer一个一个地更新vRouter
   ->将一个vRouter移至新的vRouter时，新的controller也将获得k8s-default-pod-network的路由目标（route-target），并且容器之间仍然可以ping通正常工作（ist.py ctr路由摘要以及ping的结果将在稍后附上）
6.将所有vRouter移至新的controller后，停止批处理作业、旧controller上的kube-manager和webui
    之后，继续执行ISSU进程，新controller上启动批处理作业， kube-manager和webui
   ->这个阶段从开始到结束，你都无法手动更改config-database，因此可能需要一些维护时间
   （整个过程可能会持续5至15分钟，可以ping通，但是直到启动新的kube-manager时，新容器的创建才能正常工作）
7.最后，停止旧的节点上的control、config和config-database

更新vRouters时，我使用了controller的provider: bms-maint，k8s_master和vRouter，它们都已经更改为新的，以避免由于容器重启而造成干扰。我附上了原始instances.yaml和更新vRouter的instances.yaml，以便大家获取更多详细信息。

我还将在每个阶段附加ist.py ctr nei和ist.py ctr路由摘要的结果，以说明发生的相关细节。

  • 请注意，在此示例中，我实际上并未更新模块，因为此设置主要是为了突出ISSU进程（因为即使模块版本相同，ansible-deployer也会重新创建vrouter-agent容器，同时即使完成实际的模块更新，丢包的数量也不会有太大不同。）
old-controller: 172.31.19.25
new-controller: 172.31.13.9
two-vRouters: 172.31.25.102, 172.31.33.175

开始issu之前：

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei
Introspect Host: 172.31.19.25
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                   | peer_address  | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-25-102.local | 172.31.25.102 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-33-175.local | 172.31.33.175 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]# 
[root@ip-172-31-13-9 ~]# 
[root@ip-172-31-13-9 ~]# 
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei
Introspect Host: 172.31.13.9
[root@ip-172-31-13-9 ~]# 
 -> iBGP is not configured yet

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr route summary
Introspect Host: 172.31.19.25
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 7        | 7     | 2             | 5               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-pod-network | 7        | 7     | 4             | 3               | 0                |
| :k8s-default-pod-network.inet.0                    |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-service-    | 7        | 7     | 1             | 6               | 0                |
| network:k8s-default-service-network.inet.0         |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
[root@ip-172-31-13-9 ~]# 
[root@ip-172-31-13-9 ~]# 
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr route summary
Introspect Host: 172.31.13.9
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 0        | 0     | 0             | 0               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-pod-network | 0        | 0     | 0             | 0               | 0                |
| :k8s-default-pod-network.inet.0                    |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-service-    | 0        | 0     | 0             | 0               | 0                |
| network:k8s-default-service-network.inet.0         |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
[root@ip-172-31-13-9 ~]# 
 -> 在新的控制器里没有导入路由

[root@ip-172-31-19-25 contrail-ansible-deployer]# kubectl get pod -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE                                               NOMINATED NODE
cirros-deployment-75c98888b9-6qmcm   1/1     Running   0          4m58s   10.47.255.249   ip-172-31-25-102.ap-northeast-1.compute.internal   <none>
cirros-deployment-75c98888b9-lxq4k   1/1     Running   0          4m58s   10.47.255.250   ip-172-31-33-175.ap-northeast-1.compute.internal   <none>
[root@ip-172-31-19-25 contrail-ansible-deployer]# 

/ # ip -o a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000\    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue \    link/ether 02:6b:dc:98:ac:95 brd ff:ff:ff:ff:ff:ff
13: eth0    inet 10.47.255.249/12 scope global eth0\       valid_lft forever preferred_lft forever
/ # ping 10.47.255.250
PING 10.47.255.250 (10.47.255.250): 56 data bytes
64 bytes from 10.47.255.250: seq=0 ttl=63 time=2.155 ms
64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.904 ms
^C
--- 10.47.255.250 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.904/1.529/2.155 ms
/ # 
 -> 两个vRouter每个都有一个容器，在两个容器之间ping的结果正常。

在provision_control之后:

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei
Introspect Host: 172.31.19.25
+------------------------+---------------+----------+----------+-----------+-------------+-----------------+------------+-----------+
| peer                   | peer_address  | peer_asn | encoding | peer_type | state       | send_state      | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+-----------------+------------+-----------+
| ip-172-31-13-9.local   | 172.31.13.9   | 64512    | BGP      | internal  | Idle        | not advertising | 0          | n/a       |
| ip-172-31-25-102.local | 172.31.25.102 | 0        | XMPP     | internal  | Established | in sync         | 0          | n/a       |
| ip-172-31-33-175.local | 172.31.33.175 | 0        | XMPP     | internal  | Established | in sync         | 0          | n/a       |
+------------------------+---------------+----------+----------+-----------+-------------+-----------------+------------+-----------+
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei
Introspect Host: 172.31.13.9
[root@ip-172-31-13-9 ~]#
 -> iBGP 在老的控制器上，但是新的控制器还没有那些配置（在执行pre-sync后，这会被复制到新的控制器上）

在run-sync之后:
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei
Introspect Host: 172.31.19.25
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                   | peer_address  | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-13-9.local   | 172.31.13.9   | 64512    | BGP      | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-25-102.local | 172.31.25.102 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-33-175.local | 172.31.33.175 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei
Introspect Host: 172.31.13.9
+-----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                  | peer_address | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+-----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-19-25.local | 172.31.19.25 | 64512    | BGP      | internal  | Established | in sync    | 0          | n/a       |
+-----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]#
 -> iBGP 建立起来了，ctr route summary没有改变，因为新的控制器没有k8s-default-pod-network的路由目标（route-target），路由目标（route target）过滤组织了导入这些前缀。

在迁移节点到新的controller之后:

/ # ping 10.47.255.250
PING 10.47.255.250 (10.47.255.250): 56 data bytes
64 bytes from 10.47.255.250: seq=0 ttl=63 time=1.684 ms
64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.835 ms
64 bytes from 10.47.255.250: seq=2 ttl=63 time=0.836 ms
(snip)
64 bytes from 10.47.255.250: seq=37 ttl=63 time=0.878 ms
64 bytes from 10.47.255.250: seq=38 ttl=63 time=0.823 ms
64 bytes from 10.47.255.250: seq=39 ttl=63 time=0.820 ms
64 bytes from 10.47.255.250: seq=40 ttl=63 time=1.364 ms
64 bytes from 10.47.255.250: seq=44 ttl=63 time=2.209 ms
64 bytes from 10.47.255.250: seq=45 ttl=63 time=0.869 ms
64 bytes from 10.47.255.250: seq=46 ttl=63 time=0.857 ms
64 bytes from 10.47.255.250: seq=47 ttl=63 time=0.855 ms
64 bytes from 10.47.255.250: seq=48 ttl=63 time=0.845 ms
64 bytes from 10.47.255.250: seq=49 ttl=63 time=0.842 ms
64 bytes from 10.47.255.250: seq=50 ttl=63 time=0.885 ms
64 bytes from 10.47.255.250: seq=51 ttl=63 time=0.891 ms
64 bytes from 10.47.255.250: seq=52 ttl=63 time=0.909 ms
64 bytes from 10.47.255.250: seq=53 ttl=63 time=0.867 ms
64 bytes from 10.47.255.250: seq=54 ttl=63 time=0.884 ms
64 bytes from 10.47.255.250: seq=55 ttl=63 time=0.865 ms
64 bytes from 10.47.255.250: seq=56 ttl=63 time=0.840 ms
64 bytes from 10.47.255.250: seq=57 ttl=63 time=0.877 ms
^C
--- 10.47.255.250 ping statistics ---
58 packets transmitted, 55 packets received, 5% packet loss
round-trip min/avg/max = 0.810/0.930/2.209 ms
/ # 
 -> 在vrouter-agent 重启后，可以看到丢了3个包(序号40-44)。在迁移vRouter到新的之后，ping工作地很好。

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei
检查主机：172.31.19.25
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                   | peer_address  | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-13-9.local   | 172.31.13.9   | 64512    | BGP      | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-33-175.local | 172.31.33.175 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei
Introspect Host: 172.31.13.9
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                   | peer_address  | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-19-25.local  | 172.31.19.25  | 64512    | BGP      | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-25-102.local | 172.31.25.102 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]# 
 -> 两个控制器具有XMPP连接，建立了IBGP

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr route summary
检查主机: 172.31.19.25
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 7        | 7     | 1             | 6               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-pod-network | 7        | 7     | 1             | 6               | 0                |
| :k8s-default-pod-network.inet.0                    |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-service-    | 7        | 7     | 0             | 7               | 0                |
| network:k8s-default-service-network.inet.0         |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr route summary
检查主机: 172.31.13.9
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 7        | 7     | 1             | 6               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-pod-network | 7        | 7     | 3             | 4               | 0                |
| :k8s-default-pod-network.inet.0                    |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-service-    | 7        | 7     | 1             | 6               | 0                |
| network:k8s-default-service-network.inet.0         |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
[root@ip-172-31-13-9 ~]# 
 -> 因为两个控制器具有至少一个容器来自k8s-default-pod-network, 它们使用iBGP来交换前缀，因此它们具有同一个前缀

在将第二个vrouter迁移到新的控制器之后:
/ # ping 10.47.255.250
PING 10.47.255.250 (10.47.255.250): 56 data bytes
64 bytes from 10.47.255.250: seq=0 ttl=63 time=1.750 ms
64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.815 ms
64 bytes from 10.47.255.250: seq=2 ttl=63 time=0.851 ms
64 bytes from 10.47.255.250: seq=3 ttl=63 time=0.809 ms
(snip)
64 bytes from 10.47.255.250: seq=34 ttl=63 time=0.853 ms
64 bytes from 10.47.255.250: seq=35 ttl=63 time=0.848 ms
64 bytes from 10.47.255.250: seq=36 ttl=63 time=0.833 ms
64 bytes from 10.47.255.250: seq=37 ttl=63 time=0.832 ms
64 bytes from 10.47.255.250: seq=38 ttl=63 time=0.910 ms
64 bytes from 10.47.255.250: seq=42 ttl=63 time=2.071 ms
64 bytes from 10.47.255.250: seq=43 ttl=63 time=0.826 ms
64 bytes from 10.47.255.250: seq=44 ttl=63 time=0.853 ms
64 bytes from 10.47.255.250: seq=45 ttl=63 time=0.851 ms
64 bytes from 10.47.255.250: seq=46 ttl=63 time=0.853 ms
64 bytes from 10.47.255.250: seq=47 ttl=63 time=0.851 ms
64 bytes from 10.47.255.250: seq=48 ttl=63 time=0.855 ms
64 bytes from 10.47.255.250: seq=49 ttl=63 time=0.869 ms
64 bytes from 10.47.255.250: seq=50 ttl=63 time=0.833 ms
64 bytes from 10.47.255.250: seq=51 ttl=63 time=0.859 ms
64 bytes from 10.47.255.250: seq=52 ttl=63 time=0.866 ms
64 bytes from 10.47.255.250: seq=53 ttl=63 time=0.840 ms
64 bytes from 10.47.255.250: seq=54 ttl=63 time=0.841 ms
64 bytes from 10.47.255.250: seq=55 ttl=63 time=0.854 ms
^C
--- 10.47.255.250 ping statistics ---
56 packets transmitted, 53 packets received, 5% packet loss
round-trip min/avg/max = 0.799/0.888/2.071 ms
/ #
 -> 3 packet loss is seen (seq 38-42)

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei
Introspect Host: 172.31.19.25
+----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                 | peer_address | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-13-9.local | 172.31.13.9  | 64512    | BGP      | internal  | Established | in sync    | 0          | n/a       |
+----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei
Introspect Host: 172.31.13.9
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                   | peer_address  | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-19-25.local  | 172.31.19.25  | 64512    | BGP      | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-25-102.local | 172.31.25.102 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-33-175.local | 172.31.33.175 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]# 
 -> 新的控制器具有两个XMPP连接。

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr route summary
检查主机：172.31.19.25
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 0        | 0     | 0             | 0               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-pod-network | 0        | 0     | 0             | 0               | 0                |
| :k8s-default-pod-network.inet.0                    |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-service-    | 0        | 0     | 0             | 0               | 0                |
| network:k8s-default-service-network.inet.0         |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr route summary
检查主机：172.31.13.9
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 7        | 7     | 2             | 5               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-pod-network | 7        | 7     | 4             | 3               | 0                |
| :k8s-default-pod-network.inet.0                    |          |       |               |                 |                  |
| default-domain:k8s-default:k8s-default-service-    | 7        | 7     | 1             | 6               | 0                |
| network:k8s-default-service-network.inet.0         |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
[root@ip-172-31-13-9 ~]#
 -> 老的控制器不再具有前缀。

在ISSU过程结束后，新的kube-manager 启动：
[root@ip-172-31-19-25 ~]# kubectl get pod -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE
cirros-deployment-75c98888b9-6qmcm    1/1     Running   0          34m   10.47.255.249   ip-172-31-25-102.ap-northeast-1.compute.internal   <none>
cirros-deployment-75c98888b9-lxq4k    1/1     Running   0          34m   10.47.255.250   ip-172-31-33-175.ap-northeast-1.compute.internal   <none>
cirros-deployment2-648b98685f-b8pxw   1/1     Running   0          15s   10.47.255.247   ip-172-31-25-102.ap-northeast-1.compute.internal   <none>
cirros-deployment2-648b98685f-nv7z9   1/1     Running   0          15s   10.47.255.248   ip-172-31-33-175.ap-northeast-1.compute.internal   <none>
[root@ip-172-31-19-25 ~]# 
 -> 通过新的IP创建容器（10.47.255.247, 10.47.255.248 是来自新控制器的新地址）

[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei
Introspect Host: 172.31.19.25
+----------------------+--------------+----------+----------+-----------+--------+-----------------+------------+-----------------------------+
| peer                 | peer_address | peer_asn | encoding | peer_type | state  | send_state      | flap_count | flap_time                   |
+----------------------+--------------+----------+----------+-----------+--------+-----------------+------------+-----------------------------+
| ip-172-31-13-9.local | 172.31.13.9  | 64512    | BGP      | internal  | Active | not advertising | 1          | 2019-Jun-23 05:37:02.614003 |
+----------------------+--------------+----------+----------+-----------+--------+-----------------+------------+-----------------------------+
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei
Introspect Host: 172.31.13.9
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer                   | peer_address  | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-25-102.local | 172.31.25.102 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
| ip-172-31-33-175.local | 172.31.33.175 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-13-9 ~]#
 -> 新控制器不再有iBGP路由到旧的控制器。旧控制器依然具有iBGP路由条目，虽然这个过程很快就要停止:)

在停止了就的控制器后，配置：
[root@ip-172-31-19-25 ~]# kubectl get pod -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE
cirros-deployment-75c98888b9-6qmcm    1/1     Running   0          48m   10.47.255.249   ip-172-31-25-102.ap-northeast-1.compute.internal   <none>
cirros-deployment-75c98888b9-lxq4k    1/1     Running   0          48m   10.47.255.250   ip-172-31-33-175.ap-northeast-1.compute.internal   <none>
cirros-deployment2-648b98685f-b8pxw   1/1     Running   0          13m   10.47.255.247   ip-172-31-25-102.ap-northeast-1.compute.internal   <none>
cirros-deployment2-648b98685f-nv7z9   1/1     Running   0          13m   10.47.255.248   ip-172-31-33-175.ap-northeast-1.compute.internal   <none>
cirros-deployment3-68fb484676-ct9q9   1/1     Running   0          18s   10.47.255.245   ip-172-31-25-102.ap-northeast-1.compute.internal   <none>
cirros-deployment3-68fb484676-mxbzq   1/1     Running   0          18s   10.47.255.246   ip-172-31-33-175.ap-northeast-1.compute.internal   <none>
[root@ip-172-31-19-25 ~]# 
 -> 新容器依然可以被创建

[root@ip-172-31-25-102 ~]# contrail-status 
Pod      Service  Original Name           State    Id            Status         
vrouter  agent    contrail-vrouter-agent  running  9a46a1a721a7  Up 33 minutes  
vrouter  nodemgr  contrail-nodemgr        running  11fb0a7bc86d  Up 33 minutes  

vrouter kernel module is PRESENT
== Contrail vrouter ==
nodemgr: active
agent: active

[root@ip-172-31-25-102 ~]# 
 -> 具有新控制器的vRouter工作良好

/ # ping 10.47.255.250
PING 10.47.255.250 (10.47.255.250): 56 data bytes
64 bytes from 10.47.255.250: seq=0 ttl=63 time=1.781 ms
64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.857 ms
^C
--- 10.47.255.250 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.857/1.319/1.781 ms
/ #
 -> 在vRouter之间Ping成功了

向后兼容

由于有好几种更新集群的方法（就地、ISSU、是否为ifdown vhost0），因此方法的选择也是个重要的话题。

在讨论细节之前，让我先描述一下vrouter-agent up / down的行为，以及ifup vhost0 / ifdown vhost0的行为。

重新启动vrouter-agent时，一种假设是重新创建了vrouter-agent容器和vhost0。

实际上，事实并非如此，因为vhost0与vrouter.ko是紧密耦合的，需要从kernel中卸载vrouter.ko的同时将其删除。所以从操作角度来说，需要ifdown vhost0，那么不仅需要更新vrouter-agent，还需要更新vrouter.ko。（ifdown vhost0也将在内部执行rmmod vrouter）。

因此，要讨论向后兼容，需要研究下面三个主题。

  1. controller与vrouter-agent的兼容性
  • 如果没有向后兼容性，则需要ISSU
  1. vrouter-agent与vrouter.ko的兼容性
  • 如果没有向后兼容性，则需要ifdown vhost0，这将导致最少5-10秒的流量损失，因此实际上意味着需要将流量转移到其它节点，如实时迁移（live migration）
  • 由于vrouter-agent使用netlink与vrouter.ko同步数据，因此架构更改可能导致vrouter-agent发生意外行为（例如Ksync logic上的vrouter-agent分段错误）
  1. vrouter.ko与kernel的兼容性

  • 如果没有向后兼容性，则需要更新kernel，因此这意味着需要将流量移至其它节点

  • 当vrouter.ko具有不同的in-kernal API时，无法由kernel加载，并且无法创建vhost0和vrouter-agent

对于2和3，基于种种原因不可避免地需要进行kernel更新，因此一个可行的计划是，首先选择一个新的kernel版本，然后选择一个支持该kernel的vrouter-agent / vrouter.ko，再并检查当前使用的vrouter-agent是否可以与该版本的control一起使用。

  • 如果运行良好，请使用就地更新；如果由于某些原因无法运行，或者需要回滚操作，此时需要使用ISSU

对于1，由于在导入config-api定义时ifmap会为每个版本维护white_list。

根据我的尝试，它似乎具有不错的向后兼容性（由于路由信息更新与BGP类似，因此大多数情况下也应该可以正常工作）。

为了验证这一点，我尝试使用不同版本的模块进行该设置，看起来仍然可以正常工作。

I-1. config 2002-latest, control 2002-latest, vrouter 5.0-latest, openstack queens

I-2. config 2002-latest, control 5.0-latest, vrouter 5.0-latest, openstack queens

II-1. config 2002-latest, control 2002-latest, vrouter r5.1, kubernetes 1.12

注意：不幸的是，这种组合不能很好地工作（cni无法从vrouter-agent获取端口信息），我想这是由于5.0.x和5.1之间的cni版本更改（0.2.0-> 0.3.1）引起的。

II-2. config 2002-latest, control 2002-latest, vrouter 5.0-latest, kubernetes 1.12

因此，即使不需要立即更改kernel和vRouter版本，比较频繁地更新config / control也是一个好习惯，以修复可能的错误。

Tungsten Fabric入门宝典系列文章——

1.首次启动和运行指南

  1. TF组件的七种“武器”
  2. 编排器集成
  3. 关于安装的那些事（上）
  4. 关于安装的那些事（下）
  5. 主流监控系统工具的集成
  6. 开始第二天的工作
  7. 8个典型故障及排查Tips

Tungsten Fabric 架构解析系列文章——

第一篇：TF主要特点和用例
第二篇：TF怎么运作
第三篇：详解vRouter体系结构
第四篇：TF的服务链
第五篇：vRouter的部署选项
第六篇：TF如何收集、分析、部署？
第七篇：TF如何编排
第八篇：TF支持API一览
第九篇：TF如何连接到物理网络
第十篇：TF基于应用程序的安全策略

Tungsten Fabric入门宝典丨关于集群更新的那些事
Tungsten Fabric入门宝典丨关于集群更新的那些事

猜你喜欢

转载自blog.51cto.com/14638699/2497599
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
周排行
循环神经网络（rnn）讲解
Tigao教程四：单独的关节运动
金蝶K3WISE15.0-注册套打教程
如何在Mac上配置Kubernetes
Android应用结束自身进程的方法
SpringMVC学习 十三 拦截器栈
中国驻洛杉矶总领馆举行新春招待会
HttpClient get post 发送
11 - three.js 笔记 - 绘制三维字体模型
Mysql递归获取某个父节点下面的所有子节点和子节点上的所有父节点
每日归档
更多
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022