big picture

Learn about the different networking options supported by Calico so you can choose the best one for your needs.

value

Calico's flexible, modular architecture supports a wide range of deployment options, so you can choose the networking method that best fits your specific environment and needs. This includes the ability to operate in non-overlay or overlay mode with various CNI and IPAM plugins and underlying network types, with or without BGP.

concept

If you want to get a comprehensive overview of the networking options available to you, we recommend that you make sure you are familiar with and understand the following concepts. If you want to skip the learning and go straight to options and advice, you can skip to the web option.

Kubernetes Networking Fundamentals

The Kubernetes networking model defines a "flat" network where:

Each pod has its own IP address.
Pods on any node can communicate with all pods on all other nodes without NAT.

This creates a clean, backward-compatible model where Pods can be treated as virtual machines or physical hosts from the perspective of port allocation, naming, service discovery, load balancing, application configuration, and migration. Network segmentation can be defined using network policies to restrict traffic within these basic network functions.

In this model, there is a great deal of flexibility to support different networking methods and environments. The exact details of the network implementation depend on the combination of CNI, network and cloud provider plugins used.

CNI plugin

CNI (Container Networking Interface) is a standard API that allows different networking implementations to be plugged into Kubernetes. Kubernetes calls the API whenever a pod is created or destroyed. There are two types of CNI plugins:

CNI network plugin: Responsible for adding or removing pods in the Kubernetes pod network. This includes creating/deleting each pod's network interface and connecting/disconnecting it from the rest of the networking implementation.
CNI IPAM Plugin: Responsible for allocating and releasing IP addresses for Pods when they are created or deleted. Depending on the plugin, this might include assigning each node one or more IP address (CIDR) ranges, or obtaining IP addresses from the underlying public cloud network to assign to pods.

Cloud Provider Integration

The Kubernetes Cloud Provider Integration is a cloud-specific controller that configures the underlying cloud network to help provision the Kubernetes network. Depending on the cloud provider, this may include automatically programming routing into the underlying cloud network so that it natively knows how to route pod traffic.

Benet

Kubenet is a very basic networking plugin built into Kubernetes. It does not implement cross-node networking or network policies. It is typically used with cloud provider integrations to set up routing within the cloud provider network to communicate between nodes or in a single node environment. Kubenet is not compatible with Calico.

Overlay networks

An overlay network is a network layered on top of another network. In the context of Kubernetes, an overlay network can be used to handle pod-to-pod traffic between nodes on top of an underlay network that has no knowledge of pod IP addresses or which pods are running on which nodes. An overlay network works by encapsulating network packets that the underlying network does not know how to handle (such as using pod IP addresses) in external packets that the underlying network does know how to handle (such as node IP addresses). Two common networking protocols used for encapsulation are VXLAN and IP-in-IP.

The main advantages of using an overlay network

It is that it reduces the dependence on the underlying network. For example, you can run a VXLAN overlay on top of almost any underlay network without integrating with or making any changes to the underlay network.

The main disadvantages of using an overlay network are:

Minor performance impact. The process of encapsulating the packet is CPU-intensive, and the extra bytes required in the packet to encode the encapsulation (VXLAN or IP-in-IP header) reduces the maximum size of the inner packet that can be sent, which in turn can mean that Send more packets for the same amount of total data.
Pod IP addresses are not routable outside the cluster. More on this below!

Override across subnets

In addition to standard VXLAN or IP-in-IP overlays, Calico supports VXLAN and IP-in-IP "cross-subnet" modes. In this mode, in each subnet, the underlying network acts as the L2 network. Packets sent within a single subnet are not encapsulated, so you get the performance of a non-overlay network. Packets sent across subnets are encapsulated just like a normal overlay network, reducing dependence on the underlying network (without integrating with or making any changes to the underlying network).

Just like a standard overlay network, the underlay network is unaware of pod IP addresses, and pod IP addresses cannot be routed outside the cluster.

Pod IP routable outside the cluster

An important differentiating characteristic of different Kubernetes networking implementations is whether pod IP addresses are routable outside the cluster through the wider network.

not routable

If the pod IP address is not routable outside the cluster, then when the pod tries to establish a network connection to the IP address outside the cluster, Kubernetes uses a technique called SNAT (Source Network Address Translation) to change the source IP from the pod's IP address to The IP address of the node hosting the pod. Any return packets on the connection are automatically mapped back to the pod IP address. So the pod is unaware that SNAT is happening, the destination of the connection sees the node as the source of the connection, and the underlying wider network never sees the pod IP address.

For connections in the opposite direction, something outside the cluster needs to connect to a pod, which can only be done through a Kubernetes service or a Kubernetes ingress. Nothing outside the cluster can connect directly to the Pod IP address because the wider network doesn't know how to route packets to the Pod IP address.

routable

If pod IP addresses are routable outside the cluster, pods can connect to the outside world without SNAT, and the outside world can connect directly to pods without going through Kubernetes services or Kubernetes ingress.

The advantages of Pod IP addresses that are routable outside the cluster are:

Avoiding SNAT for outbound connections may be essential for integration with existing broader security requirements. It also simplifies debugging and comprehensibility of operational logs.
If you have specialized workloads, meaning that some pods need to be directly accessible without going through Kubernetes services or Kubernetes ingress, then routable pod IPs may be operationally simpler than the alternative of using host-networked pods.

The main disadvantage of Pod IP addresses that are routable outside the cluster is that Pod IPs must be unique within the wider network. For example, if you run multiple clusters, you will need to use a different IP address range (CIDR) for the Pods in each cluster. This in turn can lead to the challenge of IP address range exhaustion when operating at scale, or if the business has other significant existing needs for IP address space.

What determines routability?

If you use an overlay network for your cluster, pod IPs are generally not routable outside the cluster.

If you are not using an overlay network, whether the pod IP is routable outside the cluster depends on the CNI plugin being used, cloud provider integration, or (for on-premises) a combination of BGP and physical network peering.

BGP

BGP (Border Gateway Protocol) is a standards-based networking protocol for sharing routes in a network. It is one of the basic building blocks of the Internet and has excellent scaling properties.

Calico has built-in support for BGP. In an on-premises deployment, this allows Calico to peer with the physical network (typically with a top-of-rack router) to exchange routes, forming a non-overlay network where pod IP addresses are routable across the wider network, just like any attached The same goes for other workloads to the network.

About the Calico Network

Calico's flexible network modular architecture includes the following.

Calico CNI network plugin

The Calico CNI network plugin uses a pair of virtual Ethernet devices (veth pairs) to connect pods to L3 routes of the host network namespace. This L3 architecture avoids the unnecessary complexity and performance overhead of additional L2 bridges found in many other Kubernetes networking solutions.

Calico CNI IPAM plugin

The Calico CNI IPAM plugin assigns IP addresses to Pods from one or more configurable IP address ranges, and dynamically assigns small blocks of IPs to each node as needed. The result is more efficient use of IP address space than many other CNI IPAM plugins, including host-native IPAM plugins for many networking solutions.

overlay network mode

Calico can provide VXLAN or IP-in-IP overlay networking, including span-only modes.

non-overlay network mode

Calico can provide a non-overlay network running on top of any underlying L2 network or L3 network, either a public cloud network with appropriate cloud provider integration, or a BGP-enabled network (typically on-premises with a standard Top-of network-rack router).

Network Policy Enforcement

Calico's network policy execution engine implements all the functions of Kubernetes network policies and the extended functions of Calico network policies. This works in conjunction with Calico's built-in network mode or any other Calico-compatible network plugin and cloud provider integration.

Calico-compatible CNI plugins and cloud provider integrations

In addition to the Calico CNI plugin and built-in networking modes, Calico is also compatible with many third-party CNI plugins and cloud provider integrations.

Amazon VPC CNI

The Amazon VPC CNI plugin allocates pod IPs from the underlying AWS VPC and uses AWS elastic network interfaces to provide VPC-native pod networking (pod IPs that are routable outside the cluster). It is the default network used in Amazon EKS, using Calico to enforce network policy.

Azure CNI

The Azure CNI plugin assigns pod IPs from the underlying Azure VNET Configures Azure Virtual Networks to provide VNET native pod networking (pod IPs that are routable outside the cluster). It is the default network used in Microsoft AKS, using Calico to enforce network policy.

Azure cloud provider

Azure Cloud Provider Integration can be used as an alternative to the Azure CNI plugin. It uses the host-local IPAM CNI plugin to assign pod IPs and program the underlying Azure VNET subnets with corresponding routes. Pod IPs are only routable within the VNET subnet (which generally equates to them not being routable outside the cluster).

Google cloud provider

The Google Cloud Provider Integration uses the host-native IPAM CNI plugin to assign pod IPs and program the Google Cloud Network Alias IP ranges to provide VPC-native pod networks (pod IPs routable outside the cluster) on Google Cloud. It's the default for Google Kubernetes Engine (GKE), using Calico to enforce network policy.

Host local IPAM

The host-local CNI IPAM plugin is a common IP address management CNI plugin that assigns each node a fixed-size IP address range (CIDR) and then assigns pod IP addresses from within that range. The default address range size is 256 IP addresses (a /24), but two of these IP addresses are reserved for special purposes and not assigned to pods. The simplicity of the host-native CNI IPAM plugin makes it easy to understand, but the IP address space usage is less efficient than the Calico CNI IPAM plugin.

Flannel

Flannel routes pod traffic using static per-node CIDRs obtained from the host's local IPAM CNI plugin. Flannel provides a number of networking backends, but primarily for its VXLAN overlay backend. Calico CNI and Calico Network Policy can be combined with flannel and host-native IPAM plugins to provide policy enforcement for VXLAN networks. This combination is sometimes called a "canal".

Note: Calico now has built-in support for VXLAN, for simplicity we generally recommend using the Calico+Flannel combination in preference.

network options

local

The most common network setup for Calico on-prem is non-overlay mode using BGP to peer with a physical network (typically a top-of-rack router) to make pod IPs routable outside the cluster. (Of course, if desired, you can configure the rest of your local network to limit the scope of cluster-external pod IP routes.) This setup provides a wealth of advanced Calico features, including the ability to advertise Kubernetes service IPs (cluster IP or external IP), And the ability to control IP address management at the pod, namespace or node level to support a wide range of possibilities for integration with existing enterprise networking and security requirements.

Policy	IPAM	CNI	Overlay	Routing
Calico	Calico	Calico	No	BGP

If it is not possible to connect BGP peering to the physical network, if the cluster is in a single L2 network, you can also run non-overlay mode, Calico just peers BGP between the nodes in the cluster. Although this is not strictly an overlay network, Pod IPs cannot be routed outside the cluster because the wider network does not have routes for Pod IPs.

Policy	IPAM	CNI	Overlay	Routing
Calico	Calico	Calico	No	BGP

Alternatively, you can run Calico in VXLAN or IP-in-IP overlay mode and use cross-subnet overlay mode to optimize performance within each L2 subnet.

Admired:

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	VXLAN	Calico

selective

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	IPIP	BGP

AWS

If you want pod IP addresses to be routable outside the cluster, you must use the Amazon VPC CNI plugin. This is the default network mode for EKS and Calico for network policies. Pod IP addresses are assigned from the underlying VPC, and the maximum number of pods per node depends on the instance type.

Policy	IPAM	CNI	Overlay	Routing
Calico	AWS	AWS	No	VPC Native

If you wish to avoid dependency on a specific cloud provider, or there are issues with allocating pod IPs from the underlying VPC due to IP address range exhaustion challenges, or if the maximum number of pods per node supported by the Amazon VPC CNI plugin is not sufficient for your If necessary, we recommend using Calico network in cross-subnet coverage mode. Pod IPs are not routable outside the cluster, but you can scale your cluster to the limits of Kubernetes without relying on the underlying cloud network.

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	VXLAN	BGP

You can learn more about Kubernetes networking on AWS, including how each of the above options work behind the scenes, in this short video: Everything You Need to Know About Kubernetes Networking on AWS.

Azure

If you want pod IP addresses to be routable outside the cluster, you must use the Azure CNI plugin. This is supported by AKS and Calico for network policy. Pod IP addresses are assigned from the underlying VNET.

Policy	IPAM	CNI	Overlay	Routing
Calico	Azure	Azure	No	VPC Native

If you want to use AKS but have issues assigning pod IPs from the underlying VNET due to IP address range exhaustion challenges, you can use Calico with the Azure cloud provider integration. This uses the host's local IPAM to assign /24s to each node and programs routes for these /24s in the cluster's underlying VNET subnet. Pod IPs are not routable outside of the cluster/VNET subnet, so the same Pod IP address range (CIDR) can be used across multiple clusters if desired.

Note: This is called kubenet + Calico in some AKS docs, but it's actually Calico CNI with the Azure cloud provider and doesn't use the kubenet plugin.

Policy	IPAM	CNI	Overlay	Routing
Calico	Host Local	Azure	No	VPC Native

If you are not using AKS and want to avoid dependencies on a specific cloud provider or issues with allocating pod IPs from the underlying VNET due to IP address range exhaustion challenges, we recommend using Calico networking in cross-subnet overlay mode. Pod IPs are not routable outside the cluster, but you can scale your cluster to the limits of Kubernetes without relying on the underlying cloud network.

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	VXLAN	Calico

You can learn more about Kubernetes networking on Azure, including how each of the above options work behind the scenes, in this short video: Everything You Need to Know About Kubernetes Networking on Azure.

google cloud

If you want pod IP addresses to be routable outside the cluster, you must use the Google cloud provider integration with the host-native IPAM CNI plugin. This is supported by GKE and Calico for network policy. Pod IP addresses are assigned from the underlying VPC, and the corresponding Alias IP addresses are automatically assigned to nodes.

Policy	IPAM	CNI	Overlay	Routing
Calico	Host Local	Calico	No	VPC Native

We recommend using Calico networking in overlay mode if you wish to avoid dependency on a specific cloud provider, or if allocating pod IPs from the underlying VPC is problematic due to IP address range exhaustion challenges. Since the Google Cloud network is a pure L3 network, it does not support cross-subnet mode. Pod IPs are not routable outside the cluster, but you can scale your cluster to the limits of Kubernetes without relying on the underlying cloud network.

Admired:

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	VXLAN	Calico

selective

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	IPIP	BGP

You can learn more about Kubernetes networking on Google Cloud, including how each of the above options work behind the scenes, in this short video: Everything You Need to Know About Kubernetes Networking on Google Cloud.

IBM cloud

If you're using IBM Cloud, then we recommend IKS, which has Calico built in to provide cross-subnet IP-in-IP coverage. In addition to providing network policies for pods, IKS also uses Calico network policies to protect host nodes within the cluster.

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	IPIP	BGP

anywhere

The above list of environments is obviously not exhaustive. Understanding the concepts and explanations in this guide will hopefully help you find the right approach for your environment. If you're still not sure, you can ask for advice through the Calico users' Slack or Discourse forums. Remember that if you want to get started, you can run Calico in VXLAN overlay mode in almost any environment without worrying too much about the different options.

Policy	IPAM	CNI	Cross-subnet	Routing
Calico	Calico	Calico	VXLAN	Calico

This article: https://architect.pub/projectcalico-determine-best-networking-option
Discussion: Knowledge Planet [Chief Architect Circle] or add WeChat trumpet [ca_cto] or add QQ group [792862318]
No public	【jiagoushipro】【Super Architect】 Brilliant graphic and detailed explanation of architecture methodology, architecture practice, technical principles, and technical trends. We are waiting for you, please scan and pay attention.
WeChat trumpet	[ca_cea] Community of 50,000 people, discussing: enterprise architecture, cloud computing, big data, data science, Internet of Things, artificial intelligence, security, full-stack development, DevOps, digitalization.
QQ group	[285069459] In-depth exchange of enterprise architecture, business architecture, application architecture, data architecture, technical architecture, integration architecture, security architecture. And various emerging technologies such as big data, cloud computing, Internet of Things, artificial intelligence, etc. Join the QQ group to share valuable reports and dry goods.
video number	[Super Architect] Quickly understand the basic concepts, models, methods, and experiences related to architecture in 1 minute. 1 minute a day, the structure is familiar.
knowledge planet	[Chief Architect Circle] Ask big names, get in touch with them, or get private information sharing.
Himalayas	[Super Architect] Learn about the latest black technology information and architecture experience on the road or in the car.	[Intelligent moments, Mr. Architecture will talk to you about black technology]
knowledge planet	Meet more friends, workplace and technical chat.	Knowledge Planet【Workplace and Technology】
LinkedIn	Harry	https://www.linkedin.com/in/architect-harry/
LinkedIn group	LinkedIn Architecture Group	https://www.linkedin.com/groups/14209750/
Weibo‍‍	【Super Architect】	smart moment‍
Bilibili	【Super Architect】
Tik Tok	【cea_cio】Super Architect
quick worker	【cea_cio_cto】Super Architect
little red book	[cea_csa_cto] Super Architect
website	CIO (Chief Information Officer)	https://cio.ceo
website	CIOs, CTOs and CDOs	https://cioctocdo.com
website	Architect practical sharing	https://architect.pub
website	Programmer cloud development sharing	https://pgmr.cloud
website	Chief Architect Community	https://jiagoushi.pro
网站	应用开发和开发平台	https://apaas.dev
网站	开发信息网	https://xinxi.dev
网站	超级架构师	https://jiagou.dev
网站	企业技术培训	https://peixun.dev
网站	程序员宝典	https://pgmr.pub
网站	开发者闲谈	https://blog.developer.chat
网站	CPO宝典	https://cpo.work
网站	首席安全官	https://cso.pub ‍
网站	CIO酷	https://cio.cool
网站	CDO信息	https://cdo.fyi
网站	CXO信息	https://cxo.pub