The container network plug-in so much, Why Bo cloud-based OVS depth self-study?

background

Starting in 2015, Bo cloud-based Kubernetes start and containers to help customers deliver application management platform. In the beginning, Bo cloud industry chose to use a very wide range of mature and stable calico as the default network solutions and accumulated a great deal of practical experience in terms of calico. As more and more cloud platform floor container, construction requirements on container network cloud platform part have become more sophisticated, and the number of customers we have conducted in-depth communication, although the needs vary, but conclude the main demands include :

• From the perspective of operation and maintenance management, network models tend to use the second floor: the second floor in the mainstream network data centers, subject to the needs of hardware capabilities, and the ability to manage the complexity of the operation and maintenance personnel, etc., most customers do not want BGP routing concepts such as the introduction of three, looking to adopt most of the operation and maintenance personnel are more familiar with Layer 2 network solutions.

• Cloud hope container internal network and external network interconnection: business applications tend to be deployed in the inner and outer containers cloud platform that platform inside and outside the network can directly get through, POD and VM / physical machine equal status, but also more conducive to existing cloud and seamless product integration.

• Pod need to support a fixed IP address: The following application scenarios such visits across a firewall, you need to have a fixed IP address POD. This focus on the needs of the scene appears in another application instance to access business district firewall.

• We need to manage the network and business network separation.

• IPV6 support.

• High performance, low jitter.

• Flexible network isolation: isolation and flexible hardware, including strong security software isolation.

• A network model supports both hope and Underlay Overlay: Underlay good performance, can communicate inside and outside the network; Overlay does not depend on the underlying network, flexibility, best possible support at the same time.

• We hope network model should be as simple, easy operation and maintenance management and debugging.

• Other advanced features, such as two-way speed limit, DPDK support.

Bo cloud after cloud container CNI team for the mainstream market plug conducted extensive research and found that mainstream CNI plug-in support for these requirements is not ideal, it is difficult to meet the demand at the same time as the network, embodied in the internal and external network interoperability, management business network separation, flexible network isolation mechanism, easy on the operation and maintenance management, and debugging issues.

After a comprehensive analysis of our core community needs and status of network construction, start in 18 years, based on self-development OVS depth container network plug BeyondFabric project, now the plug has been used as a container Bo cloud cloud platform focusing on two network model (calico / BeyondFabric one) as the default plug-Bo-vessel networks, supporting the stable operation for a long time more than the production system.

Layer 2 network model technology comparison

Thanks to the simplicity of standard CNI (relative to the various IETF's RFC), now a variety of plug-ins to achieve CNI can be said to be flourishing. Compare variety of CNI on the network are endless, not list them here. Because we are the main consideration in the selection phase of a Layer 2 network model (calico is already a very good solution to the three-tier), so we mainly basic unit of Layer 2 networks were compared, because the bridge capacity is too simple, so mainly ovs macvlan and comparison, and in order to facilitate comparison, cf. also be introduced calico.

As can be seen from the table, macvlan present more problems, this program may be due to relatively small minority, a lot less bug or enhanced PullRequest, so we chose to do the selection based on OVS programs. After selecting the OVS program, in order to support the above network requirements, based OVS we conducted in-depth research to enhance self, and this CNI network plug-in named BeyondFabric.

BeyondFabric

BeyondFabric Bo cloud kubernetes container network plug-ins based on self-depth study OVS CNI fully meet the standards, the use of etcd as its data storage unit, built-in sound IPAM ability to satisfy the core of the first chapter of customers mentioned in appeal.

1. BeyondFabric schematic

From the fabric of the network topology concept map you can see at a glance the cloud platform, whether network managers or business people can clearly understand a simple case of topology of the network. In this deployment model and the simplified (but also the most widely used model of) does not include complex logic controller, etc., provides a simple, efficient and stable network environment.

除了网络模型之外，图中出现的分区概念，也是博云容器云平台结合多个客户的实际使用场景，在大量生产实践中总结演化出的概念，主要面对企业多种网络业务分区以及特定业务独占Node资源的需求，后续会专门撰文介绍。分区结合网络模型，更好的体现出了客户数据中心的网络隔离的现状，是对容器云平台实际落地的又一个重要支撑。

2. BeyondFabric主要功能列表

• 同时支持VLAN（Underlay）和VXLAN（Overlay）模式

• 支持内外网互通

• 支持Pod固定IP地址

• 支持管理网络和业务网络分离

• 支持IPV6

• 高性能：网络性能接近物理网络

• 支持Kubernetes NetworkPolicy对象，可实现灵活的网络隔离机制

• 可以对网络进行可视化管理

• 支持网络双向限速

• 支持DPDK（即将发布）

3. BeyondFabric成熟度

• 大量落地案例

博云容器云平台基于BeyondFabric已经有大量的落地案例，BeondFabric在可管理性、稳定性、性能等多个方面运行良好。

• 通过kubernetes社区CNI测试套件测试

BeyondFabric完全满足CNI协议规范，我们的测试团队结合社区提供的工具和kubernetes job等网络测试套件对BeyondFabric进行了长时间的严格测试，测试结果证明BeyondFabric具备生产可用能力。

• 多种平台支持

私有云建设中，容器云平台一般运行在物理环境或vmware/openstack等虚拟化环境中。BeyondFabric对于这几种部署环境均能完善支持。对于网络环境复杂不易变更的场景下，BeyondFabric基于vxlan可以显著减少环境依赖。

4. BeyondFabric性能
BeyondFabric采用了稳定可靠的OVS作为其基本单元，所以从原理上讲其性能损耗应该是非常小的，我们在物理环境中基于万兆网络的性能测试也验证了这一点。

The green line indicates the physical bandwidth between the nodes can be used as a baseline for this test. Blue and red, respectively, represent the bandwidth test between the POD-POD and POD-NODE, baseline contrast can be seen, the performance loss of 3% or less.

5. Operation Management Tools: fabric-admin

Taking into account the level of hardware and software anomalies, such as kubelet or beyondFabirc the bug, environment (hardware damage) staff may cause to the normal operation of the system to varying degrees, so Bo cloud provides a fabric-admin tool in / opt under / cni / bin directory, its role is similar FSCK capacity of the file system, as BeyondFabric runtime management to provide a strong guarantee. While its exact match command line format kubectl, for users familiar with kubernetes very friendly.

For example, you can view the pod of IP occupancy (sample output has been truncated):

Meanwhile, fabric-admin also provides management capabilities to support a variety of run-time, you can run --help tips:

As FSCK file system is an important sign of maturity, fabric-admin project is BeyondFabric mature strong guarantee!

to sum up

The moment, the network is one of the main difficulties container cloud platform landing, BeyondFabric Bo cloud solutions in response to the industry's pain points raised, it addresses enterprise-class customers, especially for strong financial supervision needs of customers in the network a lot of pain points. Meanwhile BeyondFabric enhancement continues to provide more support for the demands of enterprise networks when landing vessel cloud platform.