微信公众号：运维开发故事，作者：华仔
引言
最近听好多朋友说，cilium很强，势必将成为主流。因其使用了ebpf，性能好，而且支持网络策略。于是，决定花点时间学习一下。在通过官网文档学习过程中，发现使用cilium作为CNI，居然可以不用安装kube-proxy了。这让我想起来，之前在面试中被问到的一个问题，面试官问我:kube-proxy是否可以不用安装，是否有其他替代品。这下不就有答案了嘛。
顺便吐槽一下，看官方文档学习，是真的有点难(毕竟全英文)；不过还是建议大家看官方文档学习，不要翻译成中文哦。那么接下来，就由我来实操一下。
环境说明
| 序号 | 事项 | 说明 |
|---|---|---|
| 1 | kubernetes version | v1.21.3 |
| 2 | cilium version | v1.10.3 |
| 3 | kubernetes安装方式 | kubeadm |
| 4 | cilium组网模式 | vxlan |
| 5 | os | ubuntu 18.04 |
| 6 | kubernetes集群规模 | 1master、2node |
正文
- 在master上初始化集群，并通过添加–skip-phases=addon/kube-proxy参数忽略kube-proxy的安装
kubeadm init --apiserver-advertise-address=10.211.55.50 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.21.3 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all --skip-phases=addon/kube-proxy
- 在两个node上执行kubeadm join，加入集群
kubeadm join 10.211.55.50:6443 --token ouez6j.02ms269v8i4psl7p --discovery-token-ca-cert-hash sha256:5fdafe0fe1adb3b60cd7bc33f033f028279a94a3944816424cc7f5bb498f6868
- 使用helm(v3)来安装cilium。先添加cilium库
helm repo add cilium https://helm.cilium.io/
- 使用如下命令安装cilium，添加kubeProxyReplacement=strict参数
helm install cilium cilium/cilium --version 1.10.3 --namespace kube-system --set kubeProxyReplacement=strict --set k8sServiceHost=10.211.55.50 --set k8sServicePort=6443
- 检查cilium安装结果
# 查看cilium agent,以daemonset方式部署在每个node节点上
root@cilium1:/# kubectl -n kube-system get pods -l k8s-app=cilium
NAME READY STATUS RESTARTS AGE
cilium-8gwg2 1/1 Running 0 8m4s
cilium-t9ffc 1/1 Running 0 8m39s
cilium-x42r6 1/1 Running 0 8m16s
# 查看cilum operator
root@cilium1:~# kubectl get po -A -o wide |grep cilium-operator
kube-system cilium-operator-5df88875-867hd 1/1 Running 5 41h 172.16.88.47 cilium3 <none> <none>
kube-system cilium-operator-5df88875-9kx8c 1/1 Running 5 41h 172.16.88.253 cilium2 <none> <none>
- 检查是否有kube-proxy组件。可以发现并没有该组件
root@cilium1:/# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
cilium-8gwg2 1/1 Running 0 10m
cilium-operator-5df88875-867hd 1/1 Running 5 27h
cilium-operator-5df88875-9kx8c 1/1 Running 5 27h
cilium-t9ffc 1/1 Running 0 11m
cilium-x42r6 1/1 Running 0 10m
coredns-59d64cd4d4-hbwg4 1/1 Running 1 27h
coredns-59d64cd4d4-l2pmt 1/1 Running 1 27h
etcd-cilium1 1/1 Running 2 27h
kube-apiserver-cilium1 1/1 Running 2 27h
kube-controller-manager-cilium1 1/1 Running 2 27h
kube-scheduler-cilium1 1/1 Running 2 27h
- 检查cilium状态，确保安装正确
root@cilium1:/# kubectl exec -n kube-system cilium-t9ffc -- cilium status
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
KVStore: Ok Disabled
Kubernetes: Ok 1.21 (v1.21.3) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [eth0 10.211.55.50 (Direct Routing)]
Cilium: Ok 1.10.3 (v1.10.3-4145278)
NodeMonitor: Listening for events on 8 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 2/254 allocated from 10.0.0.0/24,
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: BPF [eth0] 10.0.0.0/24 [IPv4: Enabled, IPv6: Disabled]
Controller Status: 20/20 healthy
Proxy Status: OK, ip 10.0.0.41, 0 redirects active on ports 10000-20000
Hubble: Ok Current/Max Flows: 817/4095 (19.95%), Flows/s: 0.95 Metrics: Disabled
Encryption: Disabled
Cluster health: 3/3 reachable (2021-08-07T15:29:05Z)
- 部署nginx来测试一下网络联通性
# nginx deployment yaml文件
cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
run: nginx
replicas: 4
template:
metadata:
labels:
run: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
# 创建nginx deployment
kubectl create -f deployment-nginx.yaml
# 查看部署结果
root@cilium1:/# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-649c4b9857-8f2v5 1/1 Running 1 26h 10.0.2.212 cilium2 <none> <none>
nginx-649c4b9857-mhsxs 1/1 Running 1 26h 10.0.1.23 cilium3 <none> <none>
nginx-649c4b9857-qw2jj 1/1 Running 1 26h 10.0.2.69 cilium2 <none> <none>
nginx-649c4b9857-vj9w2 1/1 Running 1 26h 10.0.1.126 cilium3 <none> <none>
- 创建一个nodeport service来验证service的可访问
# 创建service
kubectl expose deployment nginx --type=NodePort --port=80
# 查看service
root@cilium1:/# kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx NodePort 10.97.209.103 <none> 80:31126/TCP 26h
- 验证nodeport、cluster可访问
# 通过nodeport
root@cilium1:/# curl 127.0.0.1:31126
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
# 通过service:port
root@cilium1:/# curl 10.97.209.103
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
# 检查iptables 发现为空
root@cilium1:/# iptables-save | grep KUBE-SVC
root@cilium1:/#
# 检查ciliun service
root@cilium1:/# kubectl exec -n kube-system cilium-t9ffc -- cilium service list
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
ID Frontend Service Type Backend
1 10.96.0.1:443 ClusterIP 1 => 172.16.88.57:6443
2 10.96.0.10:9153 ClusterIP 1 => 10.0.2.229:9153
2 => 10.0.2.80:9153
3 10.96.0.10:53 ClusterIP 1 => 10.0.2.229:53
2 => 10.0.2.80:53
4 10.97.209.103:80 ClusterIP 1 => 10.0.2.69:80
2 => 10.0.1.23:80
3 => 10.0.1.126:80
4 => 10.0.2.212:80
5 172.16.88.57:31126 NodePort 1 => 10.0.2.69:80
2 => 10.0.1.23:80
3 => 10.0.1.126:80
4 => 10.0.2.212:80
6 0.0.0.0:31126 NodePort 1 => 10.0.2.69:80
2 => 10.0.1.23:80
3 => 10.0.1.126:80
4 => 10.0.2.212:80
- 从上面的安装和测试结果来，虽然我们没有安装k8s的kube-proxy组件，但是集群依然正常。说明kube-proxy组件确实是可以被替代的。
总结
以上虽然完成了kubernetes without kube-proxy的搭建和测试工作，但还是有很多事情没说明。比如使用cilium的系统要求、cilium是什么、有几种组网模式、网络策略。不过请不要着急，期待我后续的文章。