- `execution.checkpointing.interval`: Interval in milliseconds between checkpoints. During this interval, the system will generate new checkpoints
SET execution.checkpointing.interval = 6000;
- `execution.checkpointing.tolerable-failed-checkpoints`: Maximum number of consecutive failed checkpoints to allow. If the number of consecutive failed checkpoints exceeds this value, the job will fail.
SET execution.checkpointing.tolerable-failed-checkpoints = 10;
- `execution.checkpointing.timeout`: Checkpointing timeout in milliseconds. If the checkpoint operation does not complete within this time, the job will fail.
SET execution.checkpointing.timeout =600000;
- `execution.checkpointing.externalized-checkpoint-retention`: Retention policy for externalized checkpoints. `RETAIN_ON_CANCELLATION` means to retain externalized checkpoints on job cancellation.
SET execution.checkpointing.externalized-checkpoint-retention = RETAIN_ON_CANCELLATION;
- `execution.checkpointing.mode`: Checkpointing mode. `EXACTLY_ONCE` means that each checkpoint will only be generated when the job has been processed exactly once.
SET execution.checkpointing.mode = EXACTLY_ONCE;
- `execution.checkpointing.unaligned`: Whether the checkpoint is aligned. If set to `true`, checkpoints will be generated before all tasks of the job have completed.
SET execution.checkpointing.unaligned = true;
- `execution.checkpointing.max-concurrent-checkpoints`: Maximum number of concurrently generated checkpoints. No new checkpoints will be generated until this number of checkpoints have been generated.
SET execution.checkpointing.max-concurrent-checkpoints = 1;
- `state.checkpoints.num-retained`: Number of retained checkpoints. Checkpoints exceeding this number will be deleted
SET state.checkpoints.num-retained = 3;
Not used yet
SET execution.checkpointing.interval = 6000;
SET execution.checkpointing.tolerable-failed-checkpoints = 10;
SET execution.checkpointing.timeout =600000;
SET execution.checkpointing.externalized-checkpoint-retention = RETAIN_ON_CANCELLATION;
SET execution.checkpointing.mode = EXACTLY_ONCE;
SET execution.checkpointing.unaligned = true;
SET execution.checkpointing.max-concurrent-checkpoints = 1;
SET state.checkpoints.num-retained = 3;
The configuration information that works in the yaml file is as follows:
flinkConfiguration:
taskmanager.numberOfTaskSlots: "36"
state.backend: rocksdb
state.checkpoint-storage: filesystem
state.checkpoints.num-retained: "3"
state.backend.incremental: "true"
state.savepoints.dir: file:///flink-data/savepoints
state.checkpoints.dir: file:///flink-data/checkpoints
high-availability.type: kubernetes
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory # JobManager HA
high-availability.storageDir: file:///opt/flink/flink_recovery # JobManager HA数据保存路径
serviceAccount: flink
login to
kubectl -n flink exec -it session-deployment-only-taskmanager-2-1 bash
View cat flink-conf.yaml
blob.server.port: 6124
kubernetes.jobmanager.annotations: flinkdeployment.flink.apache.org/generation:2
state.checkpoints.num-retained: 3
kubernetes.jobmanager.replicas: 2
high-availability.type: kubernetes
high-availability.cluster-id: session-deployment-only
state.savepoints.dir: file:///flink-data/savepoints
kubernetes.taskmanager.cpu: 4.0
kubernetes.service-account: flink
kubernetes.cluster-id: session-deployment-only
state.checkpoint-storage: filesystem
high-availability.storageDir: file:///opt/flink/flink_recovery
kubernetes.internal.taskmanager.replicas: 1
kubernetes.container.image: localhost:5000/flink-sql:1.14.21
parallelism.default: 1
kubernetes.namespace: flink
taskmanager.numberOfTaskSlots: 36
kubernetes.rest-service.exposed.type: ClusterIP
high-availability.jobmanager.port: 6123
kubernetes.jobmanager.owner.reference: blockOwnerDeletion:false,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment,uid:fadef756-d327-4f19-b1b4-181d92c659eb,name:session-deployment-only,controller:false
taskmanager.memory.process.size: 6048m
kubernetes.internal.jobmanager.entrypoint.class: org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
kubernetes.pod-template-file: /tmp/flink_op_generated_podTemplate_4716243666145995447.yaml
state.backend.incremental: true
web.cancel.enable: false
execution.target: kubernetes-session
jobmanager.memory.process.size: 1024m
taskmanager.rpc.port: 6122
kubernetes.container.image.pull-policy: IfNotPresent
internal.cluster.execution-mode: NORMAL
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
kubernetes.jobmanager.cpu: 1.0
state.backend: rocksdb
$internal.flink.version: v1_14
state.checkpoints.dir: file:///flink-data/checkpoints
Only 3 chk files are generated
A chk-X represents a Checkpoint information, which stores Checkpoint metadata and data.
taskowned: state owned by TaskManagers
shared: shared state
By default, if the Checkpoint option is set, Flink only keeps the last successfully generated Checkpoint. When the Flink program fails, it can be recovered from the latest Checkpoint. However, if we want to keep multiple Checkpoints and be able to choose one of them for recovery according to actual needs, this will be more flexible. Flink supports keeping multiple Checkpoints. You need to add the following configuration in the Flink configuration file conf/flink-conf.yaml to specify the maximum number of Checkpoints that need to be saved. For example, specify to keep the latest 10 Checkpoints (that is, keep the above 10 chk -X):
state.checkpoints.num-retained: 10
ps1: If the Checkpoint directory is deleted, the task cannot be specified to restore from Checkpoint
ps2: If the job fails instead of manual cancel, no matter which of the above strategies is selected, the state record will be retained
ps3: RocksDB is used as the storage of incremental checkpoints, and periodic lazy merges can be performed to clear the historical state.
The final yaml configuration is as follows:
#Flink Session cluster source code please go to
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
namespace: flink
name: session-deployment-only
spec:
image: 192.168.1.249:16443/bigdata/flink-sql:1.14.21
#image: localhost:5000/flink-sql:1.14.21
flinkVersion: v1_14
#imagePullPolicy: Never # ImagePullPolicy: If there is no local
imagePullPolicy: IfNotPresent
ingress: #Ingress configuration, used to access flink web page
template : "flink.k8s.io"
className: "nginx"
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/"
flinkConfiguration:
taskmanager.numberOfTaskSlots: "48"
state.backend: rocksdb
state.checkpoint-storage: filesystem
state.checkpoints.num-retained: "20"
state.backend.incremental: "true"
state.savepoints.dir: file:///opt/flink/volume/flink-sp
state.checkpoints.dir: file:///opt/flink/volume/flink-cp
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory # JobManager HA
high-availability.storageDir: file:///opt/flink/volume/flink-ha # JobManager HA数据保存路径
serviceAccount: flink
jobManager:
replicas: 2
resource:
memory: "1024m"
cpu: 1
taskManager:
replicas: 1
resource:
memory: "6048m"
cpu: 4
podTemplate:
spec:
hostAliases:
- ip: "192.168.1.236"
hostnames:
- "sql.server"
- ip: "192.168.1.246"
hostnames:
- "doris.server"
containers:
- name: flink-main-container
env:
- name: TZ
value: Asia/Shanghai
volumeMounts:
- name: flink-volume #挂载checkpoint pvc
mountPath: /opt/flink/volume
volumes:
- name: flink-volume
persistentVolumeClaim:
claimName: flink-checkpoint-pvc
pvc：
#Flinnk checkpoint persistent storage pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: flink-checkpoint-pvc # checkpoint pvc name namespace
: flink # specifies the namespace
spec:
storageClassName: nfs-client #sc name, change to actual The sc name
accessModes:
- ReadWriteMany #Use the access mode of ReadWriteMany
resources:
requests:
storage: 20Gi #Storage capacity, change according to actual needs