flink on k8s 实践

1、Helm安装

wget https://get.helm.sh/helm-v3.11.2-linux-amd64.tar.gz

tar -zxvf helm-v3.11.2-linux-amd64.tar.gz

mv linux-amd64/helm /usr/local/bin/

helm安装好后，可以添加常用的helm源。

helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.3.1/

helm repo add stable http://mirror.azure.cn/kubernetes/charts

helm repo add bitnami https://charts.bitnami.com/bitnami

helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

2、安装cert-manager

下载 https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml

使用kubectl命令安装即可

kubectl apply -f cert-manager.yaml

kubectl get all -n cert-manager

3、Flink Kubernetes Operator安装

Flink Kubernetes Operator最简单直接的安装方式就是使用helm在线安装，命令如下：

helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.4.0/

helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator --namespace flink --create-namespace

helm -n flink ls -a

如果需要删除

helm -n flink delete flink-kubernetes-operator

kubectl get all -n flink -owide

helm list -n flink

application 模式：

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment # Flink集群在K8s的资源类型
metadata:
name: basic-example # 作业的名字
namespace: flink # 指定在flink命名空间下运行
spec:
#image: flink:1.13.6 # Flink的镜像，改为使用1.13.6

image: registry.cn-hangzhou.aliyuncs.com/cm_ns01/flink:1.14.6
flinkVersion: v1_14 # Flink的版本，与镜像版本保持一致
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
serviceAccount: flink
jobManager:
resource:
memory: "1024m"
cpu: 1
taskManager:
resource:
memory: "1024m"
cpu: 1
job:
jarURI: local:///opt/flink/flink-1.14.6/examples/streaming/StateMachineExample.jar # Flink作业的启动类所在的Jar包路径
parallelism: 2
upgradeMode: stateless

kubectl apply -f basic.yaml 报错：

kubectl delete -A ValidatingWebhookConfiguration flink-operator-flink-clusters-webhook-configuration

kubectl delete -A ValidatingWebhookConfiguration flink-operator-flink-webhook-configuration

kubectl apply -f basic.yaml

get all -n flink

kubectl get FlinkDeployment -n flink

停止作业

kubectl delete -f basic.yaml
kubectl delete FlinkDeployment basic-example -n flink

session 部署方式：

#Flink Session集群源码请到
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
namespace: flink
name: session-deployment-only
spec:
image: flink:1.14.6
flinkVersion: v1_14
imagePullPolicy: IfNotPresent # 镜像拉取策略，本地没有则从仓库拉取
ingress: # ingress配置，用于访问flink web页面
template: "flink.k8s.io"
className: "nginx"
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/"
flinkConfiguration:
taskmanager.numberOfTaskSlots: "36"
serviceAccount: flink
jobManager:
replicas: 1
resource:
memory: "1024m"
cpu: 1
taskManager:
replicas: 1
resource:
memory: "1024m"
cpu: 1

kubectl apply -f session-deployment-only.yaml

kubectl delete -f session-deployment-only.yaml

查看日志：

kubectl get pod -n flink | grep session

查看镜像，没有本地镜像仓库，需要添加

cat /etc/docker/daemon.json
{
"registry-mirrors": ["http://hub-mirror.c.163.com"],
"insecure-registries": ["http://192.168.1.249:8086","https://192.168.1.249:16443"]
}
systemctl daemon-reload

systemctl restart docker

打tag

docker tag flink14:latest 192.168.1.249:8086/bigdata/flink14:v1_14

docker push 192.168.1.249:8086/bigdata/flink14:v1_14

有权限异常报错，需要登录下

docker login 192.168.1.249:8086