This Share contents finishing dockerone community micro-channel group according to Bo-yun
Over the past few years Bo Yun landing in the enterprise cloud platform vessel encountered a lot of pain points, one more typical pain points, from the network side, I am very happy to talk about this topic with you and introduce our self-developed OVS CNI-based plug-in - Internal called fabric project.
01
Container platform networking needs of landing
Docker technology popular from about 2013 among developers up to now kubernetes has become the de facto container orchestration engine, container, micro-services, DevOps support each other and promote each other, the actual landing container case began more and more cloud platform. Especially since 2018, more and more companies start thinking about how to use the cloud platform to support its container production scenarios ultimately increase productivity.
Different from the development of test scenarios, a line of platform or system on a production scenario will require a lot stricter. Security, monitoring, processes, existing systems integration, business exposure and so must match the requirements of the building, it is impossible on the line. In this process, especially in the traditional financial enterprises to strict regulatory requirements, the containers cloud platform will encounter when landing a lot of problems, of which the most typical is a demand for construction container network cloud platform must meet the demands of the business side, operation and maintenance personnel, security personnel, network personnel.
Now most of them are based on cloud platforms container Kubernetes building, CNI plug-in is also very much available in the market, the needs of all enterprise network now there are very different, so the situation kind of network model adaptation to all customer scenarios is almost impossible. Now the mainstream is relatively mature and stable CNI such as calico in scalability, stability, good performance, but very difficult in the traditional financial enterprises floor, often need to compromise for different needs.
And the number of customers we conducted in-depth communication, although the needs vary, but conclude the main demands include:
In the mainstream two-story data center networking, capacity is limited by the hardware or management complexity, most customers do not want to introduce the concept of a three-layer routing and BGP.
Enterprise business systems are often deployed simultaneously inside and outside the vessel cloud platform that platform inside and outside the network can directly get through.
IP address is the identity of the business, hoping to have the ability to fixed IP, but is manageable, IP address auditable.
Separating management and data networks.
Have network isolation, isolated strong security hardware and software isolation flexibility are required.
Network model should be as simple, easy to control, easy to debug.
High-performance, low-jitter network throughput.
Other advanced features, such as two-way speed limit, DPDK, overlay and so on.
We conducted extensive research on the mainstream market, CNI plug, CNI found mainstream support for more "localization" of the demand is not ideal, does not meet the main points include:
Network model difference (VS three-story, of course, there are many programs L2, OVS with a flow meter, and so on advanced features, ideal for today's cloud-based environment), to adapt the current network environment and security policies.
Cloud native philosophy. Mainstream CNI better meet the concept of cloud native, but the actual needs of customers in the fact that some "anti-cloudnative" is, in fact, how to balance the general lack of practical experience between cloudnative and anti-cloudnative.
Simple and reliable. This is actually a very important point to consider, manufacturers, enterprises must have the appropriate personnel to take control of the network model, after all, after the network as the underlying cloud platform, there is a problem too big impact.
After a comprehensive analysis of our core community needs and status of network construction, beyondFabric decided to start the project, the project has been used as a container Bo cloud cloud platform focusing on two network model (calico / beyondFabric), one of the support of a number of enterprises stable production operation of the system.
02
BeyondFabric
BeyondFabric is our self-development kubernetes CNI plug, use etcd as its data storage unit, built-in sound IPAM ability to satisfy customers' core demands mentioned earlier. Because BeyondFabric is based on the design of Layer 2 networks, while doing a lot of optimization for specific needs, so that in the part of the scene (especially in the domestic attaches great importance to the financial security of enterprise data centers) performed well; but it also can not determine the BeyondFabric suitable for all scenarios, which CNI specific choice or to make an assessment based on their own circumstances. (In fact, none of the CNI to meet all the needs of the scene.)
fabric classic mode diagram of
从fabric的概念图中可以一目了然的看清楚云平台的网络拓扑，每个计算节点上安装一个OVS，并且作为一个单纯的虚拟交换机使用，容器通过veth pair连接到OVS的端口上从而自然的获得物理环境下的网络身份；在网络的层面上，容器、虚拟机、物理机是完全对等的。不论是网络管理人员还是业务人员都可以简单清晰的了解到网络的拓扑情况。而且在这种简化的部署模型中（同时也是使用度最广的模型）不包括控制器等复杂逻辑，提供了简单、高效、稳定的网络环境。
fabric的组件图
fabric是基于OVS的CNI插件，其具体职能为为POD组建网络并设置IP地址。
fabric-ctl负责网络及IP地址管理，通过RESTFUL API提供网络/IP的管理能力，如创建网络， 编辑网络，查找IP等。fabric-ctl本身是无状态的，所有状态信息存储到etcd中。
fabric-admin的主要使用人员为平台管理员或BOC运维人员，方便使用人员查看和管理网络及IPAM等。fabric-admin的命令行格式参考Kubectl。
在经典组网模式下，将ovs作为一个基本的虚拟交换机使用即可，非常简单。如果使用networkpolicy等隔离策略，需要在每个节点上引入一个分布式控制器。
网络管理能力
fabric项目除了CNI协议规定的组建容器网络的功能之外，还以restful API、annotation等方式额外提供了对网络的管理能力。通过界面集成后可以方便管理人员使用，如下图中的增加网络，查看网络，查看IP地址使用，固定IP等。
增加网络
查看网络
查看IP地址使用
固定IP地址
成熟度
接下来看一下fabric项目的成熟度，一个项目的成熟度是由很多方面决定的，除了fabric设计之初的简单网络模型，成熟的组件（无额外复杂组件，即使在做策略控制/overlay等场景下，也只是在每个节点上引入一个分布式控制器）之外，我们还做了以下几个方面的工作。
fabric-admin
考虑到软硬件层面的异常情况，例如kubelet或fabric的bug，环境（硬件损坏）等均可能对系统的正常运行造成不同程度的影响，所以提供了一个fabric-admin的工具，位于/opt/cni/bin目录下，其作用类似于文件系统的FSCK能力，为运行时管理提供保障。同时其命令行格式完全匹配kubectl，对熟悉kubernetes的用户非常友好。
例如可以查看pod的IP占用情况（示例输出已被截断） ：
同时，fabric-admin还提供了多种运行时管理能力支持，运行--help后可以提示：
如同FSCK是文件系统成熟的重要标志，fabric-admin是beyondFabric项目成熟的有力保障！（fabric-admin虽然功能强大，但客户现网环境中还从来没有被使用过，也从侧面说明了fabric项目的成熟度）
kubernetes社区CNI测试套件
因为fabric项目完全满足CNI协议规范，因此可以使用任意的CNI测试工具进行测试。我们的测试团队结合社区提供的CNI测试工具及k8s job对象等对beyondFabric进行了长时间的严格测试，测试结果证明fabric项目具备生产可用能力。
多种平台支持
私有云建设中，容器云平台一般运行在物理环境或vmware/openstack等虚拟化环境中。fabric对于这几种部署环境均能完善支持。对于网络环境复杂不易变更的场景下，fabric基于overlay可以显著减少环境依赖。
多个落地案例
博云容器云平台基于fabric已经有多个的落地案例，在可管理性、稳定性、性能等多个方面运行良好。
BeyondFabric性能
接下来看一下fabric的性能表现。由于fabric采用了稳定可靠的OVS作为其基本单元，所以从原理上讲其性能损耗应该是非常小的，我们在物理环境中基于万兆网络的性能测试也验证了这一点。图中可以看出pod-pod/pod-node的损耗较node-node约在5%左右。
博云容器云平台网络模型选型建议
然后我们来看一下选型建议。在客户落地容器云平台的过程中，我们会和客户进行大量沟通，其中一个很重要的沟通就是涉及业务方、安全人员、网络人员、运维人员的网络模型沟通。具体的选型建议如下表所示，最终的选型结果由所有涉及人员共同决定。
fabric项目的小结
OK，简单总结一下fabric项目。fabric项目解决了企业落地容器云平台的一些主要的痛点，通过经典网络模式可以很好的满足各个职能部门的诉求。但毕竟没有任何一种网络方案能满足所有的网络诉求，fabric也有其天生的缺点，例如经典网络模式下需要客户真实的IP网络，这些网络资源在容器化的环境下消耗速度很快，需要根据业务需要提前创建好网络资源，然而有些客户的IPV4资源会比较紧张。当然这一点随着VXLAN的支持和IPV6的普及，将会得到很大的改善。fabric核心是二层的解决方案，二层就必然面临扩展性的问题，我们目前的解决方案是通过分区的概念去映射真实的网络分区，然后通过扩展分区的方式扩展Kubernetes集群。
Q：IPAM的固定IP是怎么实现的？IP与Pod UID关联吗？
A：管理员录入网络信息后，Fabric会将所有IP地址存储到etcd中统一管理。目前固定IP是通过给deployment等workload对象增加Annotation实现的。IP不与Pod UID关联。
Q：这里面提到的三层网络和二层网络是指七层协议的三层二层吗？
A：是的，比如交换机工作在2层，路由器工作在三层。
Q：服务负载均衡怎么实现的呢？
A：外部流量导入集群的负载均衡是通过另外一个组件，ingress controller实现的，没有实现在CNI里面。Kubernetes svc的负载均衡是通过iptables实现的，Fabric项目也会往iptables里面加入一些规则，主要是跨节点SNAT。
Q：支持流量限流么？
A：支持Ingress/Egress限速，通过给容器加Annotation即可以实现容器的限速。
Q：有和Contiv做过对比吗？
A：选型阶段做过，比较早了，那时候貌似Contiv还不太成熟，所以没深入研究。
Q：这些网络方案有什么好的学习方式吗？
A：网络虽然很复杂，但万变不离其宗。容器网络这个词最近几年比较流行，是因为网络再容器环境下遇到了一些挑战，但网络本质的概念还是过去非常成熟的那一套。所以首先得学习基本的网络知识，然后去看下容器环境快速弹性等带来的痛点。
Q：TC怎么实现的？
A：这个实现的比较久了，早在过去重点支持Calico的时候就已经做了。有些细节模糊了，但基本是通过Linux tc实现的，因为本质是veth pair，所以限速可以在主机侧veth端实现。基本的限速命令可以查找tc机制就可以了，我们碰到限速不太准确，最后也通过调整参数解决了，误差控制在百分之几吧。
Q：与Kube-OVN做过对比吗？
A : Friends of Kube-OVN is an open source product, I understand too. First Kube-OVN and Fabric OVS projects are based on research and development, and support Overylay / underlay mode, you can achieve CNI agreement. But what makes the difference is quite large. OVN project from OpenStack, OpenStack in the network model is very heavy, concepts, components than more, OVN is also trying to unify Kubernetes / OpenStack network model, so Kube-OVN has some ability in fact no longer CNI range of spec the inside, such as load balancing, DNS, etc., in fact, the community has a corresponding implementation. The Fabric will be much simpler, is to achieve a standard of CNI, the network model is also very clear, the container can be directly integrated into the existing network environment, enterprise network management in general can control, compatibility with the existing system of safety supervision is better.
Network is very complex, very difficult to have a unified model to take into account all the scenarios, personally I think that this is Kubernetes community wise place to put these complex, not something abstract standard out to a third party to do it. It is due to the simplicity and complexity of network protocols CNI, CNI now on the market can be said to contending, their own merits. Personally I think this is a very good sign. Which CNI specific use, or to make decisions according to their own circumstances.