K8S running mode
Containerized deployment is a popular technology in the industry today, and running based on Docker images can make it easier for users to manage and maintain applications. The most popular container management tool is Kubernetes (k8s), and Flink also supports the k8s deployment mode in recent versions. The basic principle is similar to that of YARN. For specific configuration, please refer to the official website description, and we will not explain too much here.
History Server History Server
Once the cluster running the Flink job is stopped, you can only check the logs on yarn or the local disk, and you can no longer view the running Web UI before the job hangs up. It is difficult to know exactly what happened at the moment the job hung up. If we don't have Metrics monitoring, we can only analyze and locate problems through logs, so if we can restore the previous Web UI, we can find and locate some problems through the UI.
Flink provides a history server, which is used to query the statistics of completed jobs after the corresponding Flink cluster is shut down. We all know that only when the job is running, can we view the relevant WebUI statistics. Through this History Serverwe can query the statistics of these completed jobs, whether they exit normally or exit abnormally.
Additionally, it exposes a REST API that accepts HTTP requests and responds with JSON data. After the Flink task is stopped, the JobManager will archive the statistical information of the completed task, and the History Server process can query the task statistical information after the task is stopped. For example: the last Checkpoint, related configuration when the task is running.
Start the historyserver:
[atguigu@node001 flink-1.17.0]$ bin/historyserver.sh start
Starting historyserver daemon on host node001.
[atguigu@node001 flink-1.17.0]$ bin/flink run -t yarn-per-job -d -c com.atguigu.wc.WordCountStreamUnboundedDemo ../jar/FlinkTutorial-1.17-1.0-SNAPSHOT.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/flink/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
