The most detailed in the entire network - in-depth analysis of Istio Ambient Mesh traffic path

Author: Shi Zehuan

Preface

Istio Ambient Mesh is a new architecture launched by the Istio community that extracts Sidecar capabilities to ztunnel and waypoint. It also implements traffic rules under this architecture based on iptables and policy routing. Currently, there are some materials on the network that discuss the implementation of this part. There is a certain degree of analysis (such as the three series of articles launched by Solo.io), but there are still many details that have not been mentioned in any article. This article aims to provide a detailed explanation of the traffic path of Istio Ambient Mesh, and strives to present the details as clearly as possible to help readers fully understand the most critical parts of Istio Ambient Mesh. Reading this article requires readers to have basic TCP/IP protocol knowledge and operating system network knowledge, including iptables and routing. Due to space limitations, this article will not introduce these basic concepts in detail. For interested readers, it is recommended to consult the relevant information for a deeper understanding.

environment

We use the most simplified scenario to illustrate the Ambient Mesh traffic path: an HTTP request is initiated through curl from a sleep Pod running on Node-A, and the object of the request is a service named httpbin in the cluster. Under this service, there is an httpbin Pod located on Node-B. Since the service (ClusterIP) is just a virtual IP in K8s (it is not the address of any real network device), in this scenario, the request is Sent directly to the httpbin application Pod, the network path in this scenario will look like this:

However, under the influence of Ambient Mesh traffic rules, the data packets sent from the sleep Pod actually need to go through a series of iptables and policy routing and be forwarded to ztunnel or waypoint before reaching the httpbin application at the opposite end. In order to facilitate the description in the following pages, we need to first introduce all the participants in this process. Readers do not need to remember the addresses and names of these devices. They are listed here just for convenience of reference:

Complete traffic path

Although the request path of sleep -> httpbin seems very simple, under Ambient Mesh, ztunnel takes over all incoming and outgoing connections. IP packets sent from the sleep pod will be hijacked to the 15001 port of ztunnel, encrypted by ztunnel and then sent out. So in fact, the connection initiated by curl executed in the sleep pod is actually connected to the protocol stack in the ztunnel Pod. Similarly, the connection received by the httpbin Pod actually comes from the ztunnel running on the Node where it is located, so the actual connection established There are 3 TCP connections:

sleep -> NodeA ztunnel
NodeA ztunnel -> NodeB ztunnel
NodeB ztunnel -> httpbin

Since Ambient Mesh is a 4-layer transparent design, although it is forwarded by ztunnel or waypoint proxy, neither the curl application in the sleep Pod nor the httpbin application in the httpbin pod can perceive the existence of the intermediate proxy. In order to achieve this goal, data packets cannot be forwarded directly through DNAT and SNAT in many aspects, because DNAT and SNAT actions will modify the source/destination information of the data packet. These rules designed by Ambient Mesh are largely designed to ensure transparency at both ends. transmission path, in order to explain each part one by one, we need to further refine the above figure:

Let's interpret this picture. The author divides the entire path into three parts. The first part is curl to ztunnelA, represented by a red line; the second part is from ztunnelA to ztunnelB, represented by a blue line; the third part is ztunnelB to httpbin, represented by a yellow line. In addition, there are two types of lines. A line without arrows at both ends indicates that these devices are directly connected, that is, if the data packet enters from one end, it will come out from the other end. Although the connection mode of the two host eth0 devices It is different from the connection method between the Pod and the host, but this is not the focus of this article. We can also regard it as a direct connection. The focus of this article will be to explain the arrowed connection and analyze the three parts of the path separated above one by one. , clarify how the data packet is forwarded on this part of the path.

Part 1: curl to ztunnel-A

When we run curl to send a request, curl initiates a connection to the httpbin service. After DNS resolution, our connection target is the ClusterIP (not Pod) of the httpbin service. In the K8s cluster, the data packet sent to the Cluster IP will be A series of iptables rules convert the target address into a PodIP and then send it out through the host. However, in Ambient Mesh, it needs to be encrypted by ztunnel before sending it out. In this section, we will first take a look at how the data packet is routed to the ztunnel Pod. of the ztunnel process.

1）Sleep NS -> Node-A NS

Since there is only one network device eth0 in the sleep Pod, it is one end of the pod's veth pair in the pod network namespace, and its opposite end is the vethbda3de4b device in the host network namespace. The pod communicates with all external networks, whether up or down. All lines are sent and received through this channel. In Sidecar mode, in order to redirect traffic to Sidecar, there are a series of iptables rules in the pod network namespace. However, in Ambient mode, since Sidecar is no longer injected into the Pod, there are no longer any iptables rules in the Pod network namespace, so The traffic path at this stage is the same as that of a general K8s pod. Data packets sent from the Pod network protocol stack will be sent through the Pod's only network device eth0, and then reach the host's network namespace from the veth device on the host side.

2）Node-A NS -> ztunnelA NS

First uplink data packet

After the data packet reaches the host, it needs to be redirected to the ztunnel. However, the first data packet on this path (TCP handshake packet) will be forwarded to the ztunnel via a different path from the subsequent data packets. Let’s start with the first data packet. The package begins. Since the connection object initiated by curl in the sleep pod is the httpbin service, after DNS resolution, the real connection target is the ClusterIP of the httpbin service. Therefore, the source address and destination address of the three-layer data packet sent by the sleep pod protocol stack will be like this of:

K8s configures iptables rules for service routing to set the service address DNAT to a specific Endpoint address. In order to skip this logic and redirect the data packet to ztunnel, Ambient Mesh CNI adds some iptables rules for the host before the K8s rules. routing rules. After the data packet reaches the host from the sleep pod's veth, the PREROUTING chain begins to be executed. Let's take a look at the contents of the PREROUTING chain:

=== PREROUTING ===
    --- raw ---
    --- mangle ---
        -A PREROUTING -j ztunnel-PREROUTING
    --- nat ---
        -A PREROUTING -j ztunnel-PREROUTING
        -A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
        -A PREROUTING -d 172.18.0.1/32 -j DOCKER_OUTPUT

It can be seen that Istio Ambient Mesh has added rules to the mangle table and nat table to force the jump to the ztunnel-PREROUTING chain for processing. Since it jumps to the ztunnel-PREROUTING chain unconditionally, the data packets sent by the non-Ambient Mesh Pod will also be processed. Will jump to this chain for processing. In order to avoid incorrectly redirecting the traffic of Pods outside the grid to ztunnel, when the Pod is started, the CNI of Ambient Mesh will judge it. If the judgment result is that Ambient is enabled, this will be The Pod's IP is added to the ipset named ztunnel-pods-ips in the host network namespace. In the scenario of this article, the sleep Pod will be added to the ipset named ztunnel-pods-ips on Node-A, so the packets from the sleep Pod will hit the following rules in the ztunnel-PREROUTING chain: (The rules omit the rules related to the current stage irrelevant content):

=== ztunnel-PREROUTING ===
    --- raw ---
    --- mangle ---
        ......
        -A ztunnel-PREROUTING -p tcp -m set --match-set ztunnel-pods-ips src -j MARK --set-xmark 0x100/0x100
    --- nat ---
        -A ztunnel-PREROUTING -m mark --mark 0x100/0x100 -j ACCEPT
    --- filter ---

During the mangle table execution stage, since the source address (sleep Pod address) of the data packet is within ztunnel-pods-ips, this rule matches successfully and the data packet will be marked with 0x100. The data packet marked with 0x100 will be subsequently The nat table stage is accepted directly, which allows the data packet to skip the processing of K8s iptables rules. So far, the rules only put a mark on the data packet, and the destination address of the data packet has not changed. To change the routing of the data packet, you need to use the routing table. After PREROUTING ends, the data packet will enter the routing stage. Since the data packet is marked with 0x100, in the policy routing stage, the following rules will be hit (irrelevant rules are omitted):

......
101:    from all fwmark 0x100/0x100 lookup 101
......

After hitting this policy routing rule, routing table 101 will be used for this packet. Let's take a look at the contents of routing table 101:

default via 192.168.127.2 dev istioout 
10.244.2.3 dev veth51e3b96d scope link

Since the destination address of the data packet is the Cluster IP of the httpbin service, in the demonstration environment used in this article, the Cluster IP of the httpbin service is 10.96.0.105, so the default rule in this routing table will be hit. The meaning of this rule is: through The istioout device sends to the gateway 192.168.127.2. The istioout device is a bridge device, and the other end is the pistioout device in the ztunnel Pod, and its address is 192.168.127.2. After the data packet applies this routing rule, it will leave the host through the istioout device, and then reach the ztunnel Pod from the pistioout Network namespace.

After the data packet reaches the ztunnel Pod network namespace through pistioout, since the destination address of the data packet is not any network device in the ztunnel Pod, the data packet will be discarded by the protocol stack of the ztunnel Pod without any intervention. The outbound agent of the ztunnel process listens on port 15001. In order to correctly deliver the data packet to this port, some iptables rules and routing rules are also set up in the ztunnel Pod. After the data packet reaches the ztunnel Pod network namespace, it will hit the following rules :

=== PREROUTING ===
    --- raw ---
    --- mangle ---
        ......
        -A PREROUTING -i pistioout -p tcp -j TPROXY --on-port 15001 --on-ip 127.0.0.1 --tproxy-mark 0x400/0xfff
        ......
    --- nat ---
    --- filter ---

This rule redirects the destination address of the data packet entering from the pistioout device to the 15001 port of 127.0.0.1, and labels the data packet with 0x400. It should be noted that although the destination address is redirected here, tproxy will retain it. The original destination address of the data packet. The application can obtain the original destination address of the data packet through the SO_ORIGINAL_DST option of getsockopt. After the iptables rules are executed, policy routing will be executed, and the data packet will hit the following policy routing rules:

20000:  from all fwmark 0x400/0xfff lookup 100

This policy routing specifies the use of the 100 routing table for routing. Let’s take a look at the contents of the 100 routing table:

local default dev lo scope host

100 The routing table has only one rule, which is sent to the local protocol stack through the lo device. At this point, the data packet enters the ztunnel Pod local protocol stack. Since the data packet is redirected to 127.0.0.1:15001 in iptables, the data packet finally arrives. The 15001 port monitored by the ztunnel process, the path from curl to the first packet of the ztunnel process has been sorted out:

Downlink data packet

There are also some special rules for processing return packets at this stage. When the return packet enters the host through ztunnel's veth on the host, it will hit the following rules:

-A ztunnel-PREROUTING ! -s 10.244.2.3/32 -i veth51e3b96d -j MARK --set-xmark 0x210/0x210
-A ztunnel-PREROUTING -m mark --mark 0x200/0x200 -j RETURN

The meaning of this rule is: if the data packet does not come from the ztunnel Pod IP, but enters the host from the ztunnel Pod's veth, it will be marked with 0x210.

Why is this condition? Because ztunnel is actually a transparent proxy, that is to say, ztunnel is acting as the target service of the opposite end to reply to the data packet, so the source address in the data packet is the address of the opposite end service, not the address of the ztunnel Pod.

Subsequently, since the data packet is marked with 0x210, it will hit the following policy routing rules on the host during the routing phase:

100:    from all fwmark 0x200/0x200 goto 32766
......
32766:  from all lookup main

This routing rule indicates routing through the main routing table. The address of the return packet is the sleep Pod address, which is 10.244.2.8. It will hit the rule added by the K8s network in the main routing table:

10.244.2.8 dev vethbda3de4b scope host

This rule routes the data packet to the vethbda3de4b device. vethbda3de4b is the veth device of the sleep Pod on the host side. Data packets entering this device will enter the sleep Pod through the eth0 device in the sleep Pod.

It’s not over yet. After the route is determined, since the destination address of the packet is not the local machine, the iptables FORWARD chain will be executed again. There are the following rules in the mangle table of the FOWARD chain:

-A FORWARD -j ztunnel-FORWARD

It also jumps to the ztunnel-FORWARD chain unconditionally. In the ztunnel-FORWARD chain, since the data packet carries the 0x210 identifier, the following rules will be hit:

-A ztunnel-FORWARD -m mark --mark 0x210/0x210 -j CONNMARK --save-mark --nfmask 0x210 --ctmask 0x210

This rule saves the tag on the data packet to the connection. So far, this connection has also saved the 0x210 tag. When the data packet belonging to this connection passes through the host, the connection tag 0x210 will be used to match subsequent packets on this connection. All packets.

Subsequent uplink data packets

Since the first reply packet will mark the connection with 0x210, subsequent upstream packets entering the host from the sleep Pod veth will hit the following rules in the PREROUTING phase:

-A ztunnel-PREROUTING ! -i veth51e3b96d -m connmark --mark 0x210/0x210 -j MARK --set-xmark 0x40/0x40

As you can see, the connection mark is matched through the connmark module of iptables, and then the packet is marked with 0x40. The packet marked with 0x40 will hit the following rules in the subsequent routing stage:

102:    from all fwmark 0x40/0x40 lookup 102

Then use the 102 routing table for routing:

default via 10.244.2.3 dev veth51e3b96d onlink
10.244.2.3 dev veth51e3b96d scope link

This routing table is a rule created by Ambient Mesh for routing to the ztunnel Pod. Since the destination address of the data packet is the ClusterIP of the httpbin service, this stage will hit the default routing rule in the 102 routing table and send it to the ztunnel Pod through the veth51e3b96d device. After the data packet enters the veth51e3b96d device, it will enter the ztunnel Pod network namespace from the eth0 device of the ztunnel Pod, and hit the following iptables rules:

-A PREROUTING ! -d 10.244.2.3/32 -i eth0 -p tcp -j MARK --set-xmark 0x4d3/0xfff

The condition of this rule is that if the destination address of the data packet is not the ztunnel Pod address, and enters from the eth0 device, and the protocol is TCP, the data packet will be marked with a 0x4d3 mark. After marking this mark, the data packet will hit the following rules in the routing phase. :

20003:  from all fwmark 0x4d3/0xfff lookup 100

After hitting this policy route, the 100 routing table will be used:

local default dev lo scope host

This routing table routes packets to the lo device, forcing them into the native protocol stack.

Summarize

Let's briefly summarize. At this stage from Node-A to ztunnel, the first data packet leaves the host through the istioout interface, and the pistioout interface enters the ztunnel Pod. However, subsequent data packets will be marked as 0x210 because the connection is marked. Leave the host through the veth device of the ztunnel Pod, and then enter the ztunnel Pod through the eth0 device of the ztunnel Pod.

Part 2: ztunnel-A to ztunnel-B

After the data packet reaches the ztunnel, layer 4 load balancing will be performed. For this part, please refer to my article [ 1] . After layer 4 load balancing, ztunnel converts the target service IP to Pod IP. Due to the zero-trust security design, ztunnel needs to forward the data to the ztunnel of the node where the Pod IP is located first rather than the target Pod. The design of Ambient Mesh on this point is that the destination address of the data packet sent by ztunnel is the address of the target Pod (that is, httpbin Pod). In this way, the routing rules of K****8s can be used to make the data packet reach the target. The host where the Pod is located is on the opposite host, and then routes the data packet to the ztunnel Pod through a series of rules. In the next section, we will take a look at how this path is implemented.

1）ztunnel-A Networking Namespace -> Node-A Networking Namespace

After layer 4 load balancing, ztunnel will initiate a connection to port 15008 of the peer Pod. The reason why port 15008 is used (rather than the service port or the backend Pod port) is that this port is used when the iptables rule of the peer ztunnel matches. Matching conditions are made, the details of which will be discussed in Part 3. The data packets sent by ztunnel first need to reach the host from the ztunnel Pod to be forwarded to the peer Node through the host. The ztunnel Pod, like the sleep Pod, also has a pair of veth pairs connected to the host. The device name under the ztunnel Pod network namespace is eth0, the data packets entering the device will enter the host network namespace from the veth device on the host side. The destination address of the data packets sent by ztunnel is the address of the httpbin Pod, and will not hit any iptables rules in the ztunnel Pod. In routing stage, the data packet will hit the default routing rule and the data packet will be sent to the host through eth0.

2）Node-A -> Node-B

After the data packet reaches NodeA, since its destination address is the IP address of the httpbin Pod (10.244.1.17), it will hit the routing rule for the pod network segment added by K8s in the main routing table:

......
10.244.1.0/24 via 172.18.0.4 dev eth0
......

This rule indicates that the data packets destined for the 10.244.1.0/24 network segment are sent to the gateway 172.18.0.4 (Node-B address) through the eth0 device. The data packets sent by the ztunnel Pod are httpbin Pod address 10.244.1.7, so this rule will be hit. Rule, leave Node-A through the eth0 device and go to Node-B, and finally enter the NodeB network namespace through the eth0 device of NodeB:

3）Node-B Networking Namespace -> ztunnel-B Networking Namespace

First uplink data packet

On the path from the node to the ztunnel in the Inbound phase, similar to the Outbound phase, the first data packet (TCP handshake packet) will be forwarded to the ztunnel via a different path from the subsequent data packets. Also start with the analysis of the first packet. Let's take a look at Node-B's policy routing rules. Since this data packet does not carry any tags, it is not in NodeB's ztunnel-pod-ips ipset. At the same time, the destination address of the data packet is Pod. IP will not hit the iptables rules of the K8s network, so let’s look directly at the policy routing stage:

0:      from all lookup local
100:    from all fwmark 0x200/0x200 goto 32766
101:    from all fwmark 0x100/0x100 lookup 101
102:    from all fwmark 0x40/0x40 lookup 102
103:    from all lookup 100
32766:  from all lookup main
32767:  from all lookup default

Since the local routing table contains all local routing rules, it cannot be hit. The data packet does not have any tags and will not hit 100, 101, or 102. Therefore, the data packet will hit the 103 rule and use the 100 routing table. Next, let’s take a look at this route. Table contents:

10.244.1.3 dev vethbcda8cd4 scope link 
10.244.1.5 via 192.168.126.2 dev istioin src 10.244.1.1 
10.244.1.6 via 192.168.126.2 dev istioin src 10.244.1.1
10.244.1.7 via 192.168.126.2 dev istioin src 10.244.1.1

100 A routing rule is configured in the routing table for the IP address of each Pod running on this node. Each rule points to the istioin device on the node. That is to say, when using this routing table for routing, as long as the data packet If the destination address is a Pod on this machine, it is routed to the istioin device. The target address of this data packet is the httpbin Pod address 10.244.1.7. It will hit the routing rule in line 4: 10.244.1.7 via 192.168.126.2 dev istioin src 10.244.1.1, thus entering the istioin device. We mentioned the istioout device in the previous discussion. Similarly, the istioin device is used to handle inbound traffic, and the other end is the pistioin device in the ztunnel B Pod.

After the data packet enters ztunnel-B through the pistioin device, it is also because the destination address is httpbin Pod. In order not to be discarded by the ztunnel Pod, it also needs to be redirected by the iptables rule. Since the destination address of the data packet is 15008, the data The package will hit the following rules:

-A PREROUTING -i pistioin -p tcp -m tcp --dport 15008 -j TPROXY --on-port 15008 --on-ip 127.0.0.1 --tproxy-mark 0x400/0xfff

This rule matches the data packet from the pistioin device, the protocol is tcp, and the destination port is 15008, and redirects it to the 15008 port of 127.0.0.1 through TPROXY, and marks the data packet with 0x400. After the data packet is marked, The following rules will be hit during the policy routing phase:

20000:  from all fwmark 0x400/0xfff lookup 100

This policy routing rule instructs to use the 100 routing table to route packets:

local default dev lo scope host

Similar to ztunnel-A, the 100 routing table has only one routing rule, which is to route the data packet to the lo device, forcing the data packet to enter the local protocol stack. Since the destination address of the data packet is modified by TPROXY to 15008, the data packet will eventually Reach the socket that the ztunnel process is listening on port 15008. At this point, the data packet from ztunnel-A successfully reaches the ztunnel process:

Downlink data packet

When the ztunnel Pod protocol stack sends a downlink data packet, the data packet will enter the host through the eth0 device, and the following iptables rules will be hit on the host:

-A ztunnel-PREROUTING ! -s 10.244.2.3/32 -i veth51e3b96d -j MARK --set-xmark 0x210/0x210

The matching condition of this rule is that if the source address of the data packet is not the ztunnel Pod IP, but comes from the ztunnel Pod veth device, it will be marked with 0x210. Like the Outbound phase, the data packets marked with 0x210 will be routed using the main routing table:

100:    from all fwmark 0x200/0x200 goto 32766
......
32766:  from all lookup main

Since the destination address of the downlink data packet is the sleep Pod address (10.244.2.8), using the routing rules of the K8s network in the main routing table, the data packet sent to the 10.244.2.0/24 address segment will leave through the eth0 device of the host. , is routed by the router in the host network to the host where the peer pod is located.

10.244.2.0/24 via 172.18.0.3 dev eth0

After determining the route, hit the following rules in the ztunnel-FORWARD chain:

-A ztunnel-FORWARD -m mark --mark 0x210/0x210 -j CONNMARK --save-mark --nfmask 0x210 --ctmask 0x210

Just like the Outbound phase on Node-A, the connection is marked with a 0x210 mark through the reply packet. The mark on the connection will cause subsequent data packets to hit other rules.

Subsequent uplink data packets

Since the connection is marked with 0x210, subsequent data packets will hit the following iptables rules after entering Node-B:

-A ztunnel-PREROUTING ! -i vethbcda8cd4 -m connmark --mark 0x210/0x210 -j MARK --set-xmark 0x40/0x40
-A ztunnel-PREROUTING -m mark --mark 0x200/0x200 -j RETURN

Similar to Node-A, if the data packet does not come from ztunnel veth and has a 0x210 connection tag, the data packet is marked with 0x40. The purpose is to allow the data packet to be routed using the 102 routing table in the next routing stage:

102:    from all fwmark 0x40/0x40 lookup 102

102 The routing table is the routing table leading to ztunnel-B. The data packet will hit the default routing rule and be routed to vethbcda8cd4 (that is, the network device of ztunnel-B on the host), thereby entering the ztunnel Pod.

default via 10.244.1.3 dev vethbcda8cd4 onlink 
10.244.1.3 dev vethbcda8cd4 scope link

The process after entering the ztunnel Pod is the same as the Inbound stage and will not be described again here.

Summarize

The Inbound phase on Node-B is similar to the Outbound phase on Node-A. In this phase from Node-B to ztunnel, the first data packet leaves the host from the istioin interface, and the pistioin interface enters the ztunnel Pod, and subsequent data Because the connection is marked with 0x210, the packet will leave the host through the veth device of the ztunnel Pod, and then enter the ztunnel Pod through the eth0 device of the ztunnel Pod.

Part 3: ztunnel-B to httpbin

After ztunnel-B receives the connection from the peer ztunnel, it will immediately establish a TCP connection to the target Pod so that the decrypted data can be sent to the target Pod through this connection. ztunnel-B obtains it from the connection by calling getsockopt using the SO_ORIGINAL_DST option. to the original destination address, and learn the real destination port through the HTTP CONNECT handshake message sent by ztunnel-A. At the same time, in order for httpbin Pod to think that the data packet comes from sleep Pod, ztunnel needs to pass the IP_TRANSPARENT option on the sockt used to connect to httpbin. The source address of the connection is forced to be the address of the sleep Pod (ztunnel can learn the address of the sleep Pod through the source address of the connection). In this way, the source address of the data packet sent from ztunnelB is the sleep Pod address, and the destination address is the httpbin Pod address, just like the data packet is really sent from the sleep Pod. Next, we look at how this packet leaves the ztunnel Pod and eventually reaches the httpbin Pod.

1）ztunnel-B Netwoking Namespace -> Node-B Netwoking Namespace

Since the packet sent to the httpbin Pod will not hit any iptables rules in the ztunnel B Pod, the Pod will not carry any labels. The packet will hit the following policy routing in the ztunnel Pod, using the main routing table.

0:      from all lookup local
20000:  from all fwmark 0x400/0xfff lookup 100
20003:  from all fwmark 0x4d3/0xfff lookup 100
32766:  from all lookup main
32767:  from all lookup default

The contents of the main routing table are as follows:

default via 10.244.1.1 dev eth0 
10.244.1.0/24 via 10.244.1.1 dev eth0 src 10.244.1.3 
10.244.1.1 dev eth0 scope link src 10.244.1.3 
192.168.126.0/30 dev pistioin proto kernel scope link src 192.168.126.2 
192.168.127.0/30 dev pistioout proto kernel scope link src 192.168.127.2

Since the address of the httpbin Pod is 10.244.1.7, the default rule default via 10.244.1.1 dev eth0 in the routing table will be used for routing, and the data packet will be sent to the eth0 network interface to leave the ztunnel Pod and enter the host.

2）Node-B Netwoking Namespace -> httpbin Netwoking Namespace

After the data packet reaches the host, since its destination address is the address of the Pod on the local machine, it needs to avoid hitting the routing rule that redirects the data packet to the ztunnel Pod that was hit in the previous stage. Ambient Mesh uses ztunnel- The following rules have been added to the PREROUTING chain (how to jump to the ztunnel-PREROUTING chain, please refer to the previous article and will not be repeated here):

......
-A ztunnel-PREROUTING ! -s 10.244.1.3/32 -i vethbcda8cd4 -j MARK --set-xmark 0x210/0x210
......

vethbcda8cd4 is the device on the host side of the veth pair of ztunnel B Pod. After the data packet enters the eth0 device of ztunnel Pod, it reaches the host through this device. This rule matches the data packet from ztunnel Pod veth. The data packets that hit this rule will be marked with 0x210. In the second part, the data packets from ztunnel-A hit the policy routing rule numbered 103 and were routed to the ztunnel Pod using the 100 routing table. However, currently Due to the existence of the 0x210 mark, the data packet will first hit the policy routing rule numbered 100, and finally use the main routing table for routing:

0:      from all lookup local
100:    from all fwmark 0x200/0x200 goto 32766
101:    from all fwmark 0x100/0x100 lookup 101
102:    from all fwmark 0x40/0x40 lookup 102
103:    from all lookup 100
32766:  from all lookup main
32767:  from all lookup default

Let's take a look at the contents of the main routing table:

default via 172.18.0.1 dev eth0 
10.244.0.0/24 via 172.18.0.2 dev eth0 
10.244.1.2 dev veth1eb71e57 scope host 
10.244.1.3 dev vethbcda8cd4 scope host 
10.244.1.5 dev veth6cba8664 scope host 
10.244.1.6 dev vetheee0622e scope host
10.244.1.7 dev vethfc1b555e scope host
10.244.2.0/24 via 172.18.0.3 dev eth0 
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.4 
192.168.126.0/30 dev istioin proto kernel scope link src 192.168.126.1

The K8s network will insert all pod routing entries on the current node into the main routing table. The data packet sent to the Pod IP will be routed to the veth device connected to the Pod network namespace. The data packet sent to the httpbin Pod will hit The routing rule on line 7 is thus routed to the vethfc1b555e device, and then the packet will enter the httpbin Pod from the other end of the device, which is the eth0 device located within the httpbin Pod. At this point, we have completed the complete path of the data packet.

Conclusion

In this article, the author analyzes the complete path from the source Pod to the target Pod in Ambient Mesh. However, due to space limitations, there are actually some details that have not been mentioned in this article. The author will discuss the packet routing of non-pod communication in the future. Share these contents step by step. Alibaba Cloud Service Grid ASM [ 2] also officially supports Ambient mode in the just-released version 1.18. As the industry's first fully managed service grid that supports Ambient mode, ASM adapts some common network plug-ins and provides We have provided corresponding supporting documents such as routing rules and security rules in Ambient mode. Readers are welcome to come and experience it.

Related Links:

[1] This article

https://developer.aliyun.com/article/1290950

[2] Alibaba Cloud Service Grid ASM

https://account.aliyun.com/login/login.htm?oauth_callback=https%3A%2F%2Fservicemesh.console.aliyun.com%2F&lang=zh