Istio Sidecar startup sequence-caused application container network failure

1. Problems

The online application integrates the Spring Cloud K8S Config module. When the application starts, it will read the K8S ConfigMap as a configuration. Recently, it has been observed that the online application reports the following exception log when starting (occasionally, not always), indicating that the application cannot obtain the K8S ConfigMap configuration, and the configuration in the K8s ConfigMap has not actually taken effect.

2023-02-28 11:18:16.950+0800 [main] WARN  o.s.c.k.f.c.Fabric8ConfigMapPropertySource - 
`Can't read configMap with name: [my-app] in namespace: [my-app-ns]. Ignoring.`
io.fabric8.kubernetes.client.KubernetesClientException: 
`Operation: [get]  for kind: [ConfigMap]  with name: [my-app]  in namespace: [my-app-ns]  failed`.
...
Caused by: java.net.ConnectException: `Failed to connect to /10.96.0.1:443`
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)
...
Caused by: java.net.ConnectException: `Connection refused (Connection refused)`
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)	
...

After troubleshooting the K8S RBAC permission configuration problem and the K8S cluster network problem, it was finally noticed that the backend application was deployed with Istio integrated, that is, Istio Sidecar was automatically injected.
Since the containers in the pod are started sequentially according to the declaration order in spec.containers, and initContainers will be executed before all containers are started, the startup sequence of the containers is as follows:

That is, istio-init modifies the iptables in the pod to make istio-proxy take over all traffic in the pod (so that all network requests in the app need to go through istio-proxy), and if a network request is initiated during the app startup process, istio-proxy has The startup may not be completed yet, resulting in network exceptions.
That is to say, when the application reads the K8S ConfigMap, the istio-proxy has not been started yet and the network connection is rejected.
The solution to the above problems is to enable the application container to start after the istio-proxy is successfully started, so as to avoid network abnormalities.

2. Solutions for Istio 1.7 and later versions

In the change notes of istio 1.7, it is announced that the values.global.proxy.holdApplicationUntilProxyStarts configuration option is supported to support setting the sidecar to start first.
The values.global.proxy.holdApplicationUntilProxyStarts is disabled by default and can be enabled through the following settings:

--set values.global.proxy.holdApplicationUntilProxyStarts=true

2.1 Method 1: Global settings when installing Istio

# 默认使用default profile（可通过--set profile=default|demo|...调整）
istioctl install
# 设置sidecar优先启动（且sidecar启动成功后再启动其他应用容器） - 1.7新特性
--set values.global.proxy.holdApplicationUntilProxyStarts=true

2.2 Method 2: Set in the application Deployment through annotation

Note:
This method has not been actually tested yet, and there is no Istio 1.7+ environment. It is compiled according to Istio official documents.
For specific Istio official document descriptions, see:
https://istio.io/latest/docs/reference/config/istio. mesh.v1alpha1/#ProxyConfig

The core annotations are as follows:

annotations:
  # 重点：配置proxy - 设置proxy启动成功后再启动其他应用
  proxy.istio.io/config: |
    holdApplicationUntilProxyStarts: true

The specific settings are as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1
spec:
  selector:
    matchLabels:
      app: my-app
      version: v1
  replicas: 1
  template:
    metadata:
      labels:
        app: my-app
        version: v1
      annotations:
        # 开启Istio Sidecar自动注入
        sidecar.istio.io/inject: true
        # 重点：配置proxy - 设置proxy启动成功后再启动其他应用
        proxy.istio.io/config: |
          holdApplicationUntilProxyStarts: true
    spec:
      containers:
        - name: app
          image: registry/app:latest
          ports:
            - name: http-port
              containerPort: 8080
              protocol: TCP       

2.3 holdApplicationUntilProxyStarts enable effect

Enable holdApplicationUntilProxyStarts to true, the actual effect is as follows:
(1) Put the sidecar istio-proxy container first in the pod containers (that is, start it first);

(2) Set the istio-proxy lifecycle to ensure that the sidecar container is blocked until it is successfully started (blocking other containers in the pod from starting);

3. Solutions before Istio 1.7

Versions before Istio 1.7 did not provide the holdApplicationUntilProxyStarts configuration, which can be combined with the K8S lifecycle to achieve the same purpose, that is, to
block the application container and make the application container monitor the istio-proxy startup port. -proxy start ready).
For specific instructions, please refer to: https://github.com/istio/istio/issues/11130#issuecomment-506659562

The core configuration is as follows:

# 注：此处配置需结合Istio Sidecar注入一起使用，若未开启Istio Sidecar，则移除此配置
lifecycle:
  postStart:
    httpGet:
      path: /healthz/ready
      port: 15020
      scheme: HTTP

The specific settings are as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1
spec:
  selector:
    matchLabels:
      app: my-app
      version: v1
  replicas: 1
  template:
    metadata:
      labels:
        app: my-app
        version: v1
      annotations:
        # 开启Istio Sidecar自动注入
        sidecar.istio.io/inject: true
    spec:
      containers:
        - name: app
          image: registry/app:latest
          ports:
            - name: http-port
              containerPort: 8080
              protocol: TCP
          # 阻塞app容器，使app容器监听sidecar启动端口，若监听sidecar启动端口不通，则重启app容器（直到sidecar启动就绪）
          lifecycle:
            postStart:
              httpGet:
                path: /healthz/ready
                port: 15020
          

During the actual deployment, it can be observed that the log that could not read the K8S ConfigMap before will also appear. If this log appears, it means that the Istio Sidecar has not started successfully (the network is not connected), and then K8S will restart the application container. Observe the log again and read the K8S ConfigMap Success (no error log). In this way, by letting the application container monitor the Istio Sidecar Readiness Probe, the application container is forced to restart the application container itself repeatedly before the Istio Sidecar is not successfully started, so as to ensure that the Istio Sidecar has been started after the final application container is started.

---

Reference:
Several postures to control the execution order of containers in the pod
How Istio 1.7 ensures the startup order of the sidecar
Github - issues - App container unable to connect to network before sidecar is fully running #11130