Flink installation introduction
Flink has the following directories
go to conf
Enter the conf configuration file directory, the main configuration file dimensions flinkp-conf.yaml and slaves
Configure flink-conf.yaml to resolve as follows:
- jobmanager.rpc.address: localhost1 – IP address of
jobManager - jobmanager.rpc.port: 6123 – port of jobManager, default is 6123
- jobmanager.heap.mb – JVM heap size of jobManager
- taskmanager.heap.mb - jvm heap size setting
of taskManager - taskmanager.numberOfTaskSlots - number of taskSlots in taskManager, preferably set to the same number of CPUs of work nodes
- parallelism.default - number of parallel computations
- fs.default-scheme - File system source
- fs.hdfs.hadoopconf: -hdfs file path
- jobmanager.web.port - page monitoring port of jobmanager
Memory management configuration
Flink allocates taskmanager.heap.mb configuration by default and it is worth 70% to keep it for management. Memory management makes flink batch management very effective; and flink will not have the problem of OutMemoryException, because flink knows how much memory is reserved to execute the program; If the memory required by the program run by flink exceeds the memory it manages, Flink can utilize the disk; all in all, flink's memory management improves the robustness and speed of the system; the following describes the configuration file for managing memory:
taskmanager.memory.fraction – the percentage of managed memory, the default is 0.7
taskmanager.memory.size – the size of the taskManager’s specific management memory; this configuration reconfigures the taskmanager.memory.fraction configuration
taskmanager.memory.segment-size – used by the memory manager Size of memory buffer and network stack bytes
taskmanager.memory.preallocate – Whether taskmanager manages all memory when it starts
Configure the ip or hostname of the node machine in slaves slaves
Before starting flink
, configure the JDK without password and turn off the
firewall
. The script below starting flink will start a JobManager on the local node, and then connect to all worker nodes (the nodes listed in the slaves file) through SSH, and run the script on each node. Run TaskManager on it. Now your Flink system is up and running. The JobManager running on the local node will now listen and receive jobs on the configured RPC port.
web interface: localhost:8081
