This article is shared from the Huawei Cloud Community " Analysis of CrossSubnet Features in Calico IPIP Mode " by the author: You can make a friend.

Calico ipip crossSubnet mode

Calico-ipip mode and calico-bgp mode have corresponding limitations. For scenarios where some hosts are across subnets and network devices cannot use BGP, cross-subnet mode can be used to enable machines on the same subnet to use calico-BGP mode. Use calico-ipip mode across subnet machines.

The concept map is as follows:

Unified environment information:

Create k8s cluster

Create k8s installation script 1-setup-env.sh

#!/bin/bash
date
set -v

# 1.prep noCNI env
cat <<EOF | kind create cluster --name=calico-ipip-crosssubnet --image=kindest/node:v1.27.3 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: "10.98.0.0/16"
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.5.10
        node-labels: "rack=rack0"

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.5.11
        node-labels: "rack=rack0"

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.8.10
        node-labels: "rack=rack1"

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.8.11
        node-labels: "rack=rack1"

EOF

# 2.remove taints
controller_node_ip=`kubectl get node -o wide --no-headers | grep -E "control-plane" | awk -F " " '{print $6}'`
kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule-
kubectl get nodes -o wide

# 3. install tools for node
for i in $(docker ps -a --format "table {{.Names}}" |grep calico-ipip)
do
                echo $i
                docker cp /usr/bin/ping $i:/usr/bin/ping
                docker exec -it $i bash -c "apt-get -y update > /dev/null && apt-get -y install net-tools tcpdump lrzsz > /dev/null 2>&1"
done

The creation of this cluster is mainly to build a k8s cluster composed of 4 nodes, two of which are under a second-layer network. It forms a layer 2 network isolation from another group.

Cluster set up successfully

Build a clan environment

The clanb configuration is as follows: 2-setup-clab.sh

#!/bin/bash
set -v

brctl addbr br-pool0
ifconfig br-pool0 up

brctl addbr br-pool1
ifconfig br-pool1 up

cat <<EOF>clab.yaml | clab deploy -t clab.yaml -
name: calico-ipip-crosssubnet
topology:
  nodes:
    gw0:
      kind: linux
      image: burlyluo/vyos:1.4.0
      cmd: /sbin/init
      binds:
        - /lib/modules:/lib/modules
        - ./startup-conf/gw0-boot.cfg:/opt/vyatta/etc/config/config.boot
 
    br-pool0:
      kind: bridge
  
    br-pool1:
      kind: bridge

    server1:
      kind: linux
      image: burlyluo/nettool
      network-mode: container:calico-ipip-crosssubnet-control-plane
      exec:
      - ip addr add 10.1.5.10/24 dev net0
      - ip route replace default via 10.1.5.1

    server2:
      kind: linux
      image: burlyluo/nettool
      network-mode: container:calico-ipip-crosssubnet-worker
      exec:
      - ip addr add 10.1.5.11/24 dev net0
      - ip route replace default via 10.1.5.1

    server3:
      kind: linux
      image: burlyluo/nettool
      network-mode: container:calico-ipip-crosssubnet-worker2
      exec:
      - ip addr add 10.1.8.10/24 dev net0
      - ip route replace default via 10.1.8.1

    server4:
      kind: linux
      image: burlyluo/nettool
      network-mode: container:calico-ipip-crosssubnet-worker3
      exec:
      - ip addr add 10.1.8.11/24 dev net0
      - ip route replace default via 10.1.8.1
  links:
    - endpoints: ["br-pool0:br-pool0-net0", "server1:net0"]
    - endpoints: ["br-pool0:br-pool0-net1", "server2:net0"]
    - endpoints: ["br-pool1:br-pool1-net0", "server3:net0"]
    - endpoints: ["br-pool1:br-pool1-net1", "server4:net0"]

    - endpoints: ["gw0:eth1", "br-pool0:br-pool0-net2"]
    - endpoints: ["gw0:eth2", "br-pool1:br-pool1-net2"]

EOF

This configuration is actually to introduce the node network in the k8s cluster established by kind into containerLab, and then do the real network configuration based on containerLab

The gw0-boot.cfg configuration file is as follows:

interfaces {
    ethernet eth1 {
        address 10.1.5.1/24
        duplex auto
        smp-affinity auto
        speed auto
    }
    ethernet eth2 {
        address 10.1.8.1/24
        duplex auto
        smp-affinity auto
        speed auto
    }
    loopback lo {
    }
}
nat {
    source {
        rule 100 {
            outbound-interface eth0
            source {
                address 10.1.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 9600
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
                plaintext-password ""
            }
            level admin
        }
    }
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}
/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "qos@1:dhcp-server@5:webgui@1:pppoe-server@2:webproxy@2:firewall@5:pptp@1:dns-forwarding@1:mdns@1:quagga@7:webproxy@1:snmp@1:system@10:conntrack@1:l2tp@1:broadcast-relay@1:dhcp-relay@2:conntrack-sync@1:vrrp@2:ipsec@5:ntp@1:config-management@1:wanloadbalance@3:ssh@1:nat@4:zone-policy@1:cluster@1" === */
/* Release version: 1.2.8 */

containerLab networking is successful:

Deploy calico network plug-in

Although the cluster has been set up at this time, because the cni network plug-in has not been deployed, the k8s network between the cluster nodes/master is not interconnected (kubelet cannot communicate with the master).

Deploy the calico ipip crosssubnet mode network plug-in

kubectl apply -f calico.yaml
#kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.23/manifests/calico.yaml

Required: Change to CALICO_IPV4POOL_IPIP 的值 AlwaysCrossSubnet

Check the cluster node information at this time: Node status is ready

Check the cluster pod information at this time: All pods in the cluster are ready

Deploy test business verification CrossSubnet

apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
  labels:
    app: app
  name: app
spec:
  #replicas: 2
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - image: burlyluo/nettool
        name: nettoolbox
---
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  type: NodePort
  selector:
    app: app
  ports:
  - name: app
    port: 8080
    targetPort: 80
    nodePort: 32000

View business pod information:

View the routing policy on the node (master node):

It can be found that if it is going to 10.98.85.128/26an address within the network segment, it will go out through the net0 interface.

If it is an address destined for 10.98.193.192/26 or within a network segment, it will go out through the tunl0 interface; (ipip encapsulation is required) 10.98.241.128/26

Routing strategy for pod (pod on master node):

Capture packets to test the communication datapath with the subnet pod

The packet capture location is: On the net0 network card of the host where the source pod is located:

You can see in the packet capture data: the source/destination IPs are both srcpod/dstpod. The source/destination Mac are both: the net0 network card of the host machine where the source pod is located/the Mac address of the net0 network card of the host machine where the destination pod is located.

Therefore, pod communication across nodes in the same subnet does not require IPIP encapsulation and can be reached through Layer 3 routing.

Capture packets to test cross-subnet pod communication datapath

The IP addresses of the master node and worker2 node are: , there is cross-subnet node communication. 10.1.5.10/24 10.1.8.10/24

The packet capture location is the net0 network card of the host (master node) where the source pod is located:

It can be found that the packet capture information carries two layers of IP information: IPIP encapsulation is required across subnets.

The packet capture location is the tunl0 network card of the host (master node) where the source pod is located:

Click to follow and learn about Huawei Cloud’s new technologies as soon as possible~

Analysis of Cross Subnet Characteristics in Calico IPIP Mode