Rainbond design sharing series (1) Multi-tenant network design based on Midonet

The realization of Rainbond's support for multi-tenancy is based on the network support for multi-tenancy. The Rainbond public cloud requires that the network between each tenant must be isolated to form a mutually secure tenant network environment. For different SDN networks, the implementation methods are different. For example, Calico is isolated from routing rules, and Midonet can create subnets for different tenants. The bottom layer of Rainbond uses Kubernetes as the application running solution. It adopts the standard CNI network access specification, which provides standardized support for Rainbond to support various networks.

For small and medium-sized cluster users, Rainbond recommends Calico-based network solutions. As one of the common solutions in the Kubernetes community, this article will not introduce them in detail. The focus of our sharing today is for users with large clusters or strict requirements for tenant network isolation, that is, Midonet-based solutions.

What is CNI Specification

(CNI) [ https://github.com/containernetworking/cni/blob/spec-v0.2.0/SPEC.md ] (Container Networking Interface) is a general plug-in-based network for containers running on Linux The solution, originally derived from the Rkt container network design, is currently supported by project standards such as Kubernetes. The latest version has reached 0.3.0, and currently Rainbond supports version 0.2.0. It faces two abstract entities:

容器Can be considered synonymous with the Linux network namespace. The unit to which it corresponds depends on the specific container runtime implementation: for example, in an implementation that applies a container specification such as Rkt, each container runs in a unique network namespace. In Docker, on the other hand, a network namespace usually exists for each individual Docker container.

网络Refers to a set of entities that are uniquely addressable and can communicate with each other. This can be a single container (as above), a machine or other network device (such as a router). Containers can conceptually be added to or removed from one or more networks.

Each running instance of an application in Rainbond uses a network space, and the creation and destruction of running instances will set and recycle network resources respectively. Therefore, the network resource is a dynamic and reusable resource.

MidoNet SDN Network

MidoNet is a network virtualization software developed by Midkura, a Japanese SDN company. It realizes network virtualization based on the underlying physical facilities. It has the characteristics of distributed, decentralized and multi-layered. It is mainly used as the default network component in OpenStack. Virtual network solutions, especially those designed for network infrastructure, serve cloud platforms such as OpenStack and virtualize their network storage stack. MidoNet assigns a logical router to each tenant. Tenants are isolated from each other, and tenants can communicate with each other. Midonet supports L2 switching, L3 routing, L4 load balancing, stateful and stateless NAT, and logical sum Distributed firewall, BGP and ECMP support. Its architecture mainly includes the following components:

Midolman（Midonet Agent）：Midonet

Agent is installed in each computing node, responsible for establishing network flow control and providing distributed Midonet network services, routing, NAT, etc. He stores relevant virtual network information in NSDB.

Network State

Database(NSDB): Stores network configuration and status, network topology, routing, Midonet does not centrally handle network functions, it is handled by Midonet Agent, Midonet Agent will synchronize with NSDBs in real time and will synchronize and update NSDB in time when there are changes

MidonetAPI

midonet provides restful API interface to provide a complete set of model operations, which lays the foundation for us to make CNI-plugin.

MidoNet supports large-scale SDN clusters, and its architecture theoretically supports tens of thousands of nodes. Rainbond can support tens of thousands of node clusters based on Midonet network. It was originally designed for OpenStack virtual machine network. We adapt it to containers and make it a standard container network solution.

Network structure model under MidoNet multi-tenancy

SDN (Software Defined Networking), midonet software defines the network components you know well. The following briefly introduces several core software-defined concepts:

• Router (Router)
  A tenant corresponds to a Router, and the Bridge network connected to the same Router communicates with each other. Midonet creates a PrivierRouter, and all tenant routers connect to the PrivierRouter to communicate with the external network. It can be understood that a router communicates with the internal network and connects to the upper-level router to access the public network.
• Bridge (network bridge)
  There can be multiple bridges under a tenant, and each bridge uses a different network segment. For example, a bridge network segment 192.168.0.0/24can have up to 253 virtual devices connected to the bridge.
• Port (device communication port)
  The communication interface between Router and Router, and between Router and Bridge.
• Route (routing)
  routing rules, which define the rules for the forwarding port of traffic packets for the Router.
• Rule (filtering rules)
  defines packet filtering conditions. Similar to iptables.

Implementation of CNI Plug-in Based on MidoNet

The midonet data exchange works at Layer 3, but it does not provide IP address management (IPAM) itself, so the midonet-based cni plug-in needs to complete the following tasks:

IPAMDistinguish different tenants to allocate available IPs for application instances and recycle IP addresses of destroyed instances. At the same time, IP network segments need to be allocated for the routers and bridges mentioned below. Each router has an IP address, which is globally unique and does not conflict. Each Bridge has a unique network segment, and the connected virtual network card has a globally unique IP. IPAM needs data storage and is supposed to be a stateful daemon service. But CNI-Plugin must be designed as a stateless command, so we use etcd as our data storage service, and its global consistency ensures the correctness of our port assignments. 租户Router创建The first instance launch phase of a new tenant creates a virtual tenant subnet in Midonet for the current tenant. And connect to PrivierRouter and connect directly to the periphery. 租户Bridge创建A tenant can have one or more Bridges, depending on the number of instances, each bridge has a different virtual network segment, and supports up to 253 running instances. 容器网卡创建Creating a network card interface for the target container is a necessary function of the CNI plug-in. Multiple or one can be created as needed. Here, a network card is required to access the Midonet Bridge. 端到端连接建立Each Bridge needs to establish a connection with the tenant Router, and each running instance needs to establish a connection with the corresponding Bridge. 路由过滤规则Routing rules are required from the PrivierRouter to the tenant router, and routing rules and filtering rules are required from the tenant router to the Bridge.

Tenant network initialization

When a new tenant creates a container for the first time, the tenant virtual device is initialized and created. We have introduced the virtual devices that a tenant needs to create above. Here I will talk about the details. Midonet provides Rest-API to operate virtual devices. Note here that different versions of the API are used depending on which version of Midonet is used. https://github.com/barnettZQG/golang-midonetclient
We have encapsulated the midonet client based on the needs of Golang, supporting common APIs of 1.* and 5.* API versions. The creation steps are as follows:

1. Create a tenant and call the Keystone API.
2. Create a Router, including creating an incoming and outgoing Chain.
3. Create PrivierRouter Port and assign IP, create Router Port and assign IP. Create a PortLink to connect two ports.
4. Create a corresponding routing rule for the Chain created earlier
5. Create a packet filtering rule for the previously created Port
6. Create a default Bridge. And create a Port to connect to the Router.
7. Store the relevant data created above into etcd.

Container network card creation and network binding

Virtual Ethernet Pair

Referred to as veth pair, it is a pair of ports. All packets entering from one end of the pair of ports will come out from the other end, and vice versa. The two ends can exist in different network spaces (Network Namespace). After the container is successfully created, it has a network space. When the container is created, the ADD method of the CNI plugin is called to set the network. The plugin first creates a Veth pair. Put one end of it in the host network space, and call the Midonet binding API to bind it to a Port of the Bridge. The other end is in the container and is assigned and given an IP address by the IPAM module, which is consistent with the principle of the Docker0 network card.

设置容器内路由规则

Set the default route to the NIC created above. For example, the network card created above is named eth0, and the default egress route is set to the eth0 network card, so that the egress network of the user application will be managed by the Midonet network by default. In the Rainbond use case, in addition to the eth0 network card, the application requiring external network access also has a network card eth1 connected to the host, which is mapped to the host by the edge load balancing proxy to provide services to the external network, so Rainbond will also set the automatic Defined routing rules.

设置DNSSet some DNS information as needed. For example, Rainbond will set the application DNS information related to Rainbond.

Considerations for CNI Plugin Implementations

1. The add and delete operations of CNI plugins should be idempotent, that is, the same parameters passed in should have the same effect no matter how many times they are called.
2. CNI plugins should support concurrency, mainly the creation of tenant-related components and strong consistency in IP address assignment.
3. CNI plugins have certain specifications, please refer to: https://github.com/containernetworking/cni

FAQ

• What is Rainbond? Is it open source? Rainbond is the first open-source production-level serverless PaaS in China. It deeply integrates technologies such as Kubernetes-based container management, multi-type CI/CD application construction and delivery, and multi-data center resource management to provide cloud-native application lifecycle solutions. Build an interconnection ecosystem between applications and infrastructure, between applications and between infrastructures. Rainbond is currently open source based on the L-GPL (v3) open source license. Project address: https://github.com/goodrain/rainbond Official website: http://www.rainbond.com

• Is Midonet CNI-Plugin open source? The Midonet CNI-Plugin is open sourced as part of the Rainbond networking component. Project address: https://github.com/goodrain/midonet-cni