K8s（7）——kubernetes之metallb，calico，网络策略

---

前言

1、metallb负载均衡器与LoadBalancer

MetalLB官网

1.1 修改kube-proxy内容

 kubectl edit configmap -n kube-system kube-proxy      ## 修改kube-proxy内容
 
 kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'   
 ##和前面相似  重启pod生效

1.2 下两个配置清单,并将清单需要的镜像提前拉取到本地仓库

wget https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
wget https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml 
kubectl apply -f namespace.yaml
[root@server2 metallb]# ls
metallb.yaml  namespace.yaml

配置所需镜像到仓库
docker pull metallb/speaker:v0.9.5
docker pull metallb/controller:v0.9.5
docker tag metallb/controller:v0.9.5 reg.westos.org/metallb/controller:v0.9.5
docker tag metallb/speaker:v0.9.5  reg.westos.org/metallb/speaker:v0.9.5
docker push reg.westos.org/metallb/speaker:v0.9.5
docker push reg.westos.org/metallb/controller:v0.9.5

配置metallb负载均衡器
kubectl apply -f metallb.yaml
[root@server2 metallb]# kubectl get ns
NAME              STATUS   AGE
default           Active   3d10h
ingress-nginx     Active   62m
kube-node-lease   Active   3d10h
kube-public       Active   3d10h
kube-system       Active   3d10h
metallb-system    Active   9m50s


kubectl -n metallb-system get all    ## pod启动失败，因为没有创建密钥

kubectl create secret generic -n metallb-system  memberlist --from-literal=secretkey="$(openssl rand -base64 128)"  
## 创建密钥，否则pod会启动不了
kubectl -n metallb-system get secrets  ##查看创建的密钥
kubectl -n metallb-system  get pod   ## 有了密钥，pod启动成功



[root@server2 metallb]# cat config.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.25.200.100-172.25.200.200


kubectl create -f config.yaml
kubectl create -f nginx-svc.yml

配置所需镜像到仓库

测试：
可以看到自动分配地址池的IP

1.3 改造之前的ingress

  cd  ingress/
 
  kubectl get ingress
  
  vim deploy.yaml 
  
  kubectl delete -f deploy.yaml 
  kubectl apply -f deploy.yaml 
  kubectl get ns
  kubectl -n ingress-nginx get pod
  kubectl -n ingress-nginx get svc
  kubectl -n ingress-nginx get all

2、calico网络插件的使用

calico官网使用说明书

2.1 清理已运行的flanner网络插件

[root@server2 ~]# kubectl delete -f kube-flannel.yml    ##删除fannel网络插件配置
[root@server2 ~]# kubectl -n kube-system get pod    ##查看是否删除

[root@server2 ~]# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
demo                          1/1     Running   10         3d1h
deployment-6456d7c676-2s77n   1/1     Running   0          67m
deployment-6456d7c676-wx9tk   1/1     Running   0          67m
[root@server2 ~]# kubectl delete deployments.apps deployment 
[root@server2 ~]# kubectl  get svc
[root@server2 ~]# kubectl  delete svc nginx-svc 
[root@server2 ~]# kubectl get all
NAME       READY   STATUS    RESTARTS   AGE
pod/demo   1/1     Running   10         3d1h

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   4d21h


##移除fannel生成的配置文件
[root@server2 ~]# cd /etc/cni/net.d
[root@server2 net.d]# ls
10-flannel.conflist
[root@server2 net.d]# mv 10-flannel.conflist /mnt/   ##每一个节点都需要移除
[root@server3 net.d]# mv 10-flannel.conflist /mnt/
[root@server4 net.d]# mv 10-flannel.conflist /mnt/
[root@server2 net.d]# ps ax | grep gannel

2.2 将calico.yaml配置清单中的镜像拉取存入仓库

1. 拉取
[root@server1 harbor]# docker pull calico/kube-controllers:v3.16.1   ##拉取的镜像可以通过官网下载的calico.yaml文件来查看
[root@server1 harbor]# docker pull calico/cni:v3.16.1
[root@server1 harbor]# docker pull calico/pod2daemon-flexvol:v3.16.1
[root@server1 harbor]# docker pull calico/node:v3.16.1

2. 上传
[root@server1 harbor]# docker tag calico/node:v3.16.1 reg.westos.org/calico/node:v3.16.1
[root@server1 harbor]# docker push reg.westos.org/calico/node:v3.16.1 
[root@server1 harbor]# docker tag calico/pod2daemon-flexvol:v3.16.1 reg.westos.org/calico/pod2daemon-flexvol:v3.16.1
[root@server1 harbor]# docker push  reg.westos.org/calico/pod2daemon-flexvol:v3.16.1
[root@server1 harbor]# docker tag calico/cni:v3.16.1 reg.westos.org/calico/cni:v3.16.1
[root@server1 harbor]# docker push reg.westos.org/calico/cni:v3.16.1
[root@server1 harbor]# docker tag calico/kube-controllers:v3.16.1 reg.westos.org/calico/kube-controllers:v3.16.1
[root@server1 harbor]# docker push  reg.westos.org/calico/kube-controllers:v3.16.1

2.3 更换calico网络插件

[root@server2 ~]# mkdir calico   
[root@server2 ~]# cd calico/
[root@server2 calico]# ls
calico.yaml  deny-nginx.yaml
[root@server2 calico]# kubectl apply -f calico.yaml   ##应用calico网络插件
[root@server2 calico]# kubectl -n kube-system get pod    ##查看calico节点是否运行

[root@server2 calico]# cd /etc/cni/net.d/   ##查看calico插件配置，每个结点都有
[root@server2 net.d]# ls 
10-calico.conflist  calico-kubeconfig

[root@server2 calico]# cat deny-nginx.yaml 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-nginx
spec:
  podSelector:
    matchLabels:
      app: nginx

2.4 测试

1.先生成两个标签为nginx的pod
[root@server2 calico]# cd ../ingress/
[root@server2 ingress]# ls
auth  demo.yml  deploy.yaml  nginx-svc.yml  nginx.yml  tls.crt  tls.key
[root@server2 ingress]# kubectl apply -f nginx-svc.yml 
[root@server2 ingress]# kubectl get pod -L app
NAME                          READY   STATUS    RESTARTS   AGE    APP
demo                          1/1     Running   10         3d1h   
deployment-6456d7c676-plxjz   1/1     Running   0          17s    nginx
deployment-6456d7c676-rfr85   1/1     Running   0          17s    nginx
[root@server2 ingress]# kubectl get svc
[root@server2 ingress]# curl 10.96.155.46

2. 运行测试文件，查看是否隔离标签为nginx的pod(网络策略，隔离所有ngix标签的pod)
[root@server2 ~]# cd calico/
[root@server2 calico]# vim deny-nginx.yaml
[root@server2 calico]# cat deny-nginx.yaml   # NetworkPolicy模式
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy      
metadata:
  name: deny-nginx
spec:
  podSelector:
    matchLabels:
      app: nginx
[root@server2 calico]# kubectl apply -f deny-nginx.yaml   ##运行
networkpolicy.networking.k8s.io/deny-nginx created
[root@server2 calico]# kubectl get networkpolicies.networking.k8s.io   ##查看限制
NAME         POD-SELECTOR   AGE
deny-nginx   app=nginx      20s
[root@server2 calico]# kubectl  get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   4d21h
nginx-svc    ClusterIP   10.111.89.37   <none>        80/TCP    5m1s
[root@server2 calico]# curl 10.111.89.37

1.先生成两个标签为nginx的pod

3. 运行测试文件，查看是否隔离标签为nginx的pod
   

3、网络策略

参考链接

[root@server2 calico]# kubectl  -n kube-system describe pod calico-node-6k72w ##出错可以查看详细信息
[root@server2 ~]# kubectl delete -f kube-flannel.yml   ##或者出错查看日志

3.1 限制访问指定服务（和2.4相同）

# vim nginx-policy.yml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-nginx
spec:
  podSelector:
    matchLabels:
      app: nginx

3.2 允许指定pod访问服务

[root@server2 calico]# kubectl run nginx --image=myapp:v2 -it 
[root@server2 calico]# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
demo                          1/1     Running   10         3d3h
deployment-6456d7c676-plxjz   1/1     Running   0          90m
deployment-6456d7c676-rfr85   1/1     Running   0          90m
nginx                         1/1     Running   1          28s
[root@server2 calico]# kubectl get pod -o wide
NAME                          READY   STATUS    RESTARTS   AGE    IP               NODE      NOMINATED NODE   READINESS GATES
demo                          1/1     Running   10         3d3h   10.244.1.28      server3   <none>           <none>
deployment-6456d7c676-plxjz   1/1     Running   0          90m    10.244.141.192   server3   <none>           <none>
deployment-6456d7c676-rfr85   1/1     Running   0          90m    10.244.22.0        server4   <none>           <none>    ##标签为nginx的ip
nginx                         1/1     Running   1          35s    10.244.141.193   server3   <none>           <none>


[root@server2 calico]# vim access-demo.yaml
[root@server2 calico]# cat access-demo.yaml 
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: access-nginx
spec:
  podSelector:
    matchLabels:
      app: nginx
  ingress:
  - from:
      - podSelector:
          matchLabels:
            app: demo
[root@server2 calico]# kubectl apply -f access-demo.yaml 
networkpolicy.networking.k8s.io/access-nginx created
[root@server2 calico]# kubectl get networkpolicies.
NAME           POD-SELECTOR   AGE
access-nginx   app=nginx      20s
deny-nginx     app=nginx      87m

[root@server2 calico]# kubectl label pod demo app=demo   ##设置标签
pod/demo labeled
[root@server2 calico]# kubectl get pod -L app
NAME                          READY   STATUS    RESTARTS   AGE     APP
demo                          1/1     Running   11         3d3h    demo
deployment-6456d7c676-plxjz   1/1     Running   0          93m     nginx
deployment-6456d7c676-rfr85   1/1     Running   0          93m     nginx
nginx                         1/1     Running   1          3m37s   

[root@server2 calico]# kubectl attach demo -it
Defaulting container name to demo.
Use 'kubectl describe pod/demo -n default' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
/ # ping 10.244.22.0
PING 10.244.22.0 (10.244.22.0): 56 data bytes
64 bytes from 10.244.22.0: seq=0 ttl=62 time=0.812 ms
^C
--- 10.244.22.0 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.812/0.812/0.812 ms
/ # curl 10.244.22.0
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

3.3 禁止 namespace中所有pod之间的相互访问

[root@server2 calico]# kubectl create namespace demo        ##创建新的namespace实验
[root@server2 calico]# kubectl get ns
[root@server2 calico]# kubectl run  demo1 --image=busyboxplus -it -n demo ##创建demo1
[root@server2 calico]# kubectl run  demo2 --image=busyboxplus -it -n demo ##创建demo2
[root@server2 calico]# kubectl -n demo get pod
[root@server2 calico]# kubectl -n demo  get pod --show-labels 


[root@server2 calico]# vim deny-pod.yaml
[root@server2 calico]# cat deny-pod.yaml 
[root@server2 calico]# cat deny-pod.yaml 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: demo
spec:
  podSelector: {
    
    }

[root@server2 calico]# kubectl apply -f deny-pod.yaml 
[root@server2 calico]# kubectl get networkpolicies. -n demo


测试

[root@server2 calico]# kubectl  get pod -o wide -n demo  ##查看ip
NAME    READY   STATUS    RESTARTS   AGE     IP               NODE      NOMINATED NODE   READINESS GATES
demo1   0/1     Error     1          9m36s   10.244.22.2      server4   <none>           <none>
demo2   1/1     Running   1          8m32s   10.244.141.194   server3   <none>           <none>
[root@server2 calico]# kubectl attach demo1 -it -n demo   ##测试demo1和demo2之间是否可以相互访问
Defaulting container name to demo1.
Use 'kubectl describe pod/demo1 -n demo' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
/ # curl 10.244.141.194

3.4 禁止其他 namespace 访问服务

[root@server2 calico]# vim deny-ns.yaml
[root@server2 calico]# cat deny-ns.yaml 
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: deny-namespace
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {
    
    }

[root@server2 calico]# kubectl apply -f deny-ns.yaml 

[root@server2 calico]# kubectl attach demo2 -it -n demo 
Defaulting container name to demo2.
Use 'kubectl describe pod/demo2 -n demo' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
/ # curl 10.244.22.0

3.5 允许指定namespace访问服务

[root@server2 calico]# kubectl create namespace test   #创建新的ns
[root@server2 calico]# kubectl run demo3 --image=busyboxplus -it -n test   ##创建测试pod
[root@server2 calico]# vim access-ns.yaml
[root@server2 calico]# cat access-ns.yaml 
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: access-namespace
spec:
  podSelector:
    matchLabels:
      run: nginx         ##指定标签
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          role: prod    ##指定ns的角色名字

[root@server2 calico]# kubectl label ns test role=prod  ##设置namespace标签
[root@server2 calico]# kubectl get ns --show-labels 


[root@server2 calico]# kubectl apply -f access-ns.yaml   ##访问指定标签的pod
[root@server2 calico]# kubectl attach demo3 -it -n test
/ # curl 10.244.141.193    ##标签为run=nginx的pod节点，且访问成功
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

3.6 允许外网访问服务

[root@server2 ingress]# pwd
/root/ingress
[root@server2 ingress]# cat demo.yml 
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  #tls:
  #  - hosts:
  #    - www1.westos.org
  #    secretName: tls-secret
  rules:
  - host: www1.westos.org
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-svc
          servicePort: 80
[root@server2 ingress]# kubectl apply -f demo.yml 
[root@server2 ingress]# kubectl get ingress
NAME           CLASS    HOSTS             ADDRESS                   PORTS   AGE
ingress-demo   <none>   www1.westos.org   172.25.13.3,172.25.13.4   80      4h6m

[root@server2 calico]# vim access-ex.yaml
[root@server2 calico]# cat access-ex.yaml 
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-external
spec:
  podSelector:
    matchLabels:
      app: nginx
  ingress:
  - ports:
    - port: 80
    from: []