K8S external service Ingress

1. Introduction to Ingress

The role of service is reflected in two aspects. For the inside of the cluster, it continuously tracks the changes of pods, updates the objects corresponding to the pods in the endpoint (endpoint), and provides a service discovery mechanism for pods with changing IP addresses; for outside the cluster, it is similar to the load The balancer provides access to pods inside and outside the cluster.

In Kubernetes, the IP address of the Pod and the ClusterIP of the service can only be used within the cluster network and are invisible to applications outside the cluster. In order to enable external applications to access services in the cluster, Kubernetes currently provides the following solutions:

• NodePort: Exposes the service on the node network. Behind NodePort is Kube-Proxy. Kube-Proxy is the bridge that communicates with the service network, Pod network and node network.
  The test environment is fine for use, but when there are dozens or hundreds of services running in the cluster, NodePort's port management is a disaster. Because each port can only be one service, the port range can only be 30000-32767.
• LoadBalancer: Map the LoadBalancer to the LoadBalancer address provided by the cloud service provider by setting it. This usage is only used in scenarios where the Service is set up on the cloud platform of a public cloud service provider. Limited by the cloud platform, and usually deploying LoadBalancer on the cloud platform requires additional costs.
  After the service is submitted, Kubernetes will call CloudProvider to create a load balancing service for you on the public cloud, and configure the IP address of the proxied Pod to the load balancing service as the backend.
• externalIPs: The service allows external IPs to be assigned to it. If the external IP is routed to one or more Nodes in the cluster, the Service will be exposed to these externalIPs. Traffic entering the cluster through external IP will be routed to the Service Endpoint. 
• Ingress: Only one or a small number of public IPs and LBs are needed to expose multiple HTTP services to the external network at the same time, using a seven-layer reverse proxy.
  It can be simply understood as a service. It is actually a set of rules that forward user requests to one or more services based on domain names and URL paths.

1. Ingress composition

• ingress: Ingress is an API object, configured through a yaml file. The function of the ingress object is to define the rules for how requests are forwarded to the service, which can be understood as a configuration template.
  Ingress exposes cluster internal services through http or https, and provides services with external URLs, load balancing, SSL/TLS capabilities, and domain name-based reverse proxy. Ingress relies on ingress-controller to implement the above functions.
• ingress-controller:
  ingress-controller is a program that specifically implements reverse proxy and load balancing. It parses the rules defined by ingress and forwards requests according to the configured rules.
  ingress-controller is not a component that comes with k8s. In fact, ingress-controller is just a general term. Users can choose different ingress-controller implementations. Currently, the only two ingress-controllers maintained by k8s are Google Cloud's GCE and ingress-nginx. , and there are many other ingress-controllers maintained by third parties. For details, please refer to the official documentation. But no matter which type of ingress-controller, the implementation mechanism is similar, but there are differences in the specific configuration.
  Generally speaking, the ingress-controller is in the form of a pod, which runs a daemon program and a reverse proxy program. The daemon is responsible for constantly monitoring changes in the cluster, generating configurations based on the ingress object and applying new configurations to the reverse proxy. For example, ingress-nginx dynamically generates nginx configurations, dynamically updates upstream, and reloads the program to apply new configurations when needed. For convenience, the following examples take ingress-nginx officially maintained by k8s as an example.

Ingress-Nginx github address: https://github.com/kubernetes/ingress-nginx
Ingress-Nginx official website: https://kubernetes.github.io/ingress-nginx/

2. Summary

The ingress-controller is the component responsible for specific forwarding. It is exposed at the cluster entrance through various methods. External request traffic to the cluster will go to the ingress-controller first, and the ingress object is used to tell the ingress-controller how to forward the request. For example, which domain names, which URLs should be forwarded to which services, etc.

2. How Ingress works

1. Ingress-controller interacts with kubernetes APIServer to dynamically sense changes in ingress rules in the cluster.
2. Then read it and generate an nginx configuration according to the custom rules. The rules specify which domain name corresponds to which service.
3. Then write it to the pod of nginx-ingress-controller. This ingress-controller pod runs an Nginx service. The controller will write the generated nginx configuration into the /etc/nginx.conf file.
4. Then reload it to make the configuration take effect. This achieves the role of domain name distinction configuration and dynamic update.

3. Deploy nginx-ingress-controller

1. Deploy ingress-controller pod and related resources

mkdir /opt/ingress
cd /opt/ingress

官方下载地址：
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.25.0/deploy/static/mandatory.yaml

上面可能无法下载，可用国内的 gitee
wget https://gitee.com/mirrors/ingress-nginx/raw/nginx-0.25.0/deploy/static/mandatory.yaml
wget https://gitee.com/mirrors/ingress-nginx/raw/nginx-0.30.0/deploy/static/mandatory.yaml

#mandatory.yaml文件中包含了很多资源的创建，包括namespace、ConfigMap、role，ServiceAccount等等所有部署ingress-controller需要的资源。

2. Modify ClusterRole resource configuration

vim mandatory.yaml
......
apiVersion: rbac.authorization.k8s.io/v1beta1
#RBAC相关资源从1.17版本开始改用rbac.authorization.k8s.io/v1，rbac.authorization.k8s.io/v1beta1在1.22版本即将弃用
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"    # (0.25版本)增加 networking.k8s.io Ingress 资源的 api 
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"   # (0.25版本)增加 networking.k8s.io/v1 Ingress 资源的 api 
    resources:
      - ingresses/status
    verbs:
      - update

How ingress exposes services

• Method 1: Deployment+LoadBalancer mode Service
  If you want to deploy ingress in the public cloud, this method is more suitable. Use Deployment to deploy the ingress-controller and create a service of type LoadBalancer to associate with this group of pods. Most public clouds will automatically create a load balancer for the LoadBalancer service, usually bound to a public network address. As long as the domain name resolution is pointed to this address, the cluster service will be exposed to the outside world.
• 方式二：DaemonSet+HostNetwork+nodeSelector
  用DaemonSet结合nodeselector来部署ingress-controller到特定的node上，然后使用HostNetwork直接把该pod与宿主机node的网络打通，直接使用宿主机的80/433端口就能访问服务。这时，ingress-controller所在的node机器就很类似传统架构的边缘节点，比如机房入口的nginx服务器。该方式整个请求链路最简单，性能相对NodePort模式更好。缺点是由于直接利用宿主机节点的网络和端口，一个node只能部署一个ingress-controller pod。 比较适合大并发的生产环境使用。
• 方式三：Deployment+NodePort模式的Service
  同样用deployment模式部署ingress-controller，并创建对应的service，但是type为NodePort。这样，ingress就会暴露在集群节点ip的特定端口上。由于nodeport暴露的端口是随机端口，一般会在前面再搭建一套负载均衡器来转发请求。该方式一般用于宿主机是相对固定的环境ip地址不变的场景。
  NodePort方式暴露ingress虽然简单方便，但是NodePort多了一层NAT，在请求量级很大时可能对性能会有一定影响。

3、指定 nginx-ingress-controller 运行在 node02 节点

 采用方式二：DaemonSet+HostNetwork+nodeSelector

kubectl label node node02 ingress=true

kubectl get nodes --show-labels

4、修改 Deployment 为 DaemonSet ，指定节点运行，并开启 hostNetwork 网络

vim mandatory.yaml

...
apiVersion: apps/v1
# 修改 kind
# kind: Deployment
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
# 删除Replicas
# replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # 使用主机网络
      hostNetwork: true
      # 选择节点运行
      nodeSelector:
        ingress: "true"
      serviceAccountName: nginx-ingress-serviceaccount
......

5、在所有 node 节点上传 nginx-ingress-controller 镜像压缩包 ingree.contro.tar.gz 到/opt/ingress 目录，并解压和加载镜像

cd /opt/ingress
tar zxvf ingree.contro.tar.gz
docker load -i ingree.contro.tar

6、启动 nginx-ingress-controller

kubectl apply -f mandatory.yaml

#nginx-ingress-controller 已经运行 node02 节点
kubectl get pod -n ingress-nginx -o wide

kubectl get cm,daemonset -n ingress-nginx -o wide

到 node02 节点查看

netstat -lntp | grep nginx

由于配置了 hostnetwork，nginx 已经在 node 主机本地监听 80/443/8181 端口。其中 8181 是 nginx-controller 默认配置的一个 default backend(Ingress 资源没有匹配的 rule 对象时，流量就会被导向这个 default backend)。这样，只要访问 node 主机有公网 IP，就可以直接映射域名来对外网暴露服务了。如果要 nginx 高可用的话，可以在多个 node上部署，并在前面再搭建一套 LVS+keepalived 做负载均衡。

7、创建 ingress 规则

创建一个 deploy 和 svc

vim service-nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-app-svc
spec:
  type: ClusterIP
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  selector:
    app: nginx

创建 ingress

用的是方法二

#方法一：(extensions/v1beta1 Ingress 在1.22版本即将弃用)
vim ingress-app.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-app-ingress
spec:
  rules:
  - host: www.mcl.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-app-svc
          servicePort: 80

#方法二：
vim ingress-app.yaml	  
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-app-ingress
spec:
  rules:
  - host: www.mcl.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-app-svc
            port:
              number: 80

kubectl apply -f service-nginx.yaml

kubectl apply -f ingress-app.yaml

kubectl get pods

kubectl get ingress

8、测试访问

本地 host 添加域名解析

vim /etc/hosts

curl www.mcl.com

9、查看 nginx-ingress-controller

kubectl get pod -n ingress-nginx -o wide

kubectl exec -it nginx-ingress-controller-p7tdq -n ingress-nginx /bin/bash

可以看到从 start server www.mcl.com 到 end server www.mcl.com 之间包含了此域名用于反向代理的配置

四、采用方式三：Deployment+NodePort模式的Service

1、下载 nginx-ingress-controller 和 ingress-nginx 暴露端口配置文件

mkdir /opt/ingress-nodeport
cd /opt/ingress-nodeport

官方下载地址：
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/baremetal/service-nodeport.yaml

国内 gitee 资源地址：
wget https://gitee.com/mirrors/ingress-nginx/raw/nginx-0.30.0/deploy/static/mandatory.yaml
wget https://gitee.com/mirrors/ingress-nginx/raw/nginx-0.30.0/deploy/static/provider/baremetal/service-nodeport.yaml

2、在所有 node 节点上传镜像包 ingress-controller-0.30.0.tar 到 /opt/ingress-nodeport 目录，并加载镜像

tar zxvf ingree.contro-0.30.0.tar.gz
docker load -i ingress-controller-0.30.0.tar

3、启动 nginx-ingress-controller

kubectl apply -f mandatory.yaml
kubectl apply -f service-nodeport.yaml

如果K8S Pod 调度失败，在 kubectl describe pod资源时显示：
Warning  FailedScheduling  18s (x2 over 18s)  default-scheduler  0/2 nodes are available: 2 node(s) didn't match node selector

解决方案：

1. 给需要调度的node加上对应标签
   相对上面这个Yaml文件的例子
   kubectl label nodes node_name kubernetes.io/os=linux
2. 删除Yaml文件中的nodeSelector，如果对节点没有要求的话，直接删除节点选择器即可

4、Ingress HTTP 代理访问

创建 deployment、Service、Ingress Yaml 资源

vim ingress-nginx.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    name: nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-test
spec:
  rules:
  - host: www.long.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: 
            name: nginx-svc
            port:
              number: 80

kubectl get pods,svc -o wide

kubectl exec -it pod/nginx-myapp-57dd86f5cc-l48l2 bash

kubectl exec -it pod/nginx-myapp-57dd86f5cc-lpnfz bash

5、测试访问

curl 10.244.1.18

kubectl get svc -n ingress-nginx

vim /etc/hosts

192.168.160.20 master
192.168.160.40 node01
192.168.160.90 node02
192.168.160.70 hub.mcl.com
192.168.160.90 www.mcl.com
#添加域名解析
192.168.160.40 www.mcl.com www.long.com

#外部访问
curl http://www.long.com:31751

6、Ingress HTTP 代理访问虚拟主机

mkdir /opt/ingress-nodeport/vhost
cd /opt/ingress-nodeport/vhost

#创建虚拟主机1资源
vim deployment1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment1
spec:
  replicas: 2
  selector:
    matchLabels:
      name: nginx1
  template:
    metadata:
      labels:
        name: nginx1
    spec:
      containers:
        - name: nginx1
          image: soscscs/myapp:v1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: svc-1
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    name: nginx1
	
	
kubectl apply -f deployment1.yaml

#创建虚拟主机2资源
vim deployment2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment2
spec:
  replicas: 2
  selector:
    matchLabels:
      name: nginx2
  template:
    metadata:
      labels:
        name: nginx2
    spec:
      containers:
        - name: nginx2
          image: soscscs/myapp:v2
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: svc-2
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    name: nginx2
	
	
kubectl apply -f deployment2.yaml

#创建ingress资源
vim ingress-nginx.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress1
spec:
  rules:
    - host: www1.mcl.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service: 
              name: svc-1
              port:
                number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress2
spec:
  rules:
    - host: www2.mcl.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service: 
              name: svc-2
              port:
                number: 80


kubectl apply -f ingress-nginx.yaml

7、测试访问

kubectl get svc -n ingress-nginx

#做主机映射
vim /etc/hosts

curl www1.mcl.com:31751

curl www2.mcl.com:31751

8、Ingress  HTTPS 代理访问 

mkdir /opt/ingress-nodeport/https
cd /opt/ingress-nodeport/https

8.1 创建ssl证书

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"

8.2 创建 secret 资源进行存储

kubectl create secret tls tls-secret --key tls.key --cert tls.crt
kubectl get secret
kubectl describe secret tls-secret

8.3 创建 deployment、Service、Ingress Yaml 资源

vim ingress-https.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    name: nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-https
spec:
  tls:
    - hosts:
      - www3.long.com
      secretName: tls-secret
  rules:
    - host: www3.kgc.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service: 
              name: nginx-svc
              port:
                number: 80
                
kubectl apply -f ingress-https.yaml

9、Nginx 进行 BasicAuth

mkdir /opt/ingress-nodeport/basic-auth
cd　/opt/ingress-nodeport/basic-auth

9.1 生成用户密码认证文件，创建 secret 资源进行存储

yum -y install httpd
htpasswd -c auth mcl			#认证文件名必须为 auth
kubectl create secret generic basic-auth --from-file=auth
kubectl get secrets
kubectl describe secrets basic-auth

9.2 创建 ingress 资源

vim ingress-auth.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-auth
  annotations:
    #设置认证类型basic
    nginx.ingress.kubernetes.io/auth-type: basic
	#设置secret资源名称basic-auth
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
	#设置认证窗口提示信息
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - mcl'
spec:
  rules:
  - host: auth.mcl.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: 
            name: nginx-svc
            port:
              number: 80
              
//具体详细设置方法可参考官网https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

9.3 访问测试

kubectl apply -f ingress-auth.yaml
kubectl get svc -n ingress-nginx
echo '192.168.160.40 auth.mcl.com' >> /etc/hosts

浏览器访问：http://auth.mcl.com:31751

10、Nginx 进行重写

metadata.annotations 配置说明

nginx.ingress.kubernetes.io/rewrite-target: <字符串> #必须重定向流量的目标URI
nginx.ingress.kubernetes.io/ssl-redirect: <布尔值> #指示位置部分是否仅可访问SSL(当Ingress包含证书时，默认为true)
nginx.ingress.kubernetes.io/force-ssl-redirect: <布尔值> #即使Ingress未启用TLS，也强制重定向到HTTPS
nginx.ingress.kubernetes.io/app-root: <字符串> #定义Controller必须重定向的应用程序根，如果它在'/'上下文中
nginx.ingress.kubernetes.io/use-regex: <布尔值> #指示Ingress上定义的路径是否使用正则表达式

编写ingress-rewrite.yaml

vim ingress-rewrite.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-rewrite
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: http://www1.mcl.com:31751
spec:
  rules:
  - host: re.mcl.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
		  #由于re.kgc.com只是用于跳转不需要真实站点存在，因此svc资源名称可随意定义
          service: 
            name: nginx-svc
            port:
              number: 80

访问测试

kubectl apply -f ingress-rewrite.yaml
echo '192.168.160.40 re.mcl.com' >> /etc/hosts

浏览器访问：http://re.mcl.com:31751

五、总结

ingress是k8s集群的请求入口，可以理解为对多个service的再次抽象
通常说的ingress一般包括ingress资源对象及ingress-controller两部分组成
ingress-controller有多种实现，社区原生的是ingress-nginx，根据具体需求选择
ingress自身的暴露有多种方式，需要根据基础环境及业务类型选择合适的方式