Article Directory
Standalone mode
Apache Flink clusters can be deployed on Linux, Mac OS, and Windows systems. The only requirement is that Java 8.x and above are installed in the system environment.
Two 64-bit Linux systems are prepared here, and the host names of the systems are intsmaze-201 and intsmaze-202 respectively. The 64-bit JDK of version 1.8 is installed on the two machines, the firewall is turned off, and password-free login is configured for each other.
1 password-free login
Enter the "ssh-keygen" command on the intsmaze-201 node to generate a local public-private key. If an interface similar to the following appears, it means that the local public-private key is successfully generated.
[intsmaze@intsmaze-201 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/intsmaze/.ssh/id_rsa):
Create directory '/home/intsmaze/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/intsmaze/.ssh/id_rsa.
Your public key has been saved in /home/intsmaze/.ssh/id_rsa.pub.
The key fingerprint is:
32:63:ed:7c:4e:dc:ca:99:ce:b0:81:62:73:fd:a3:b9 intsmaze@intsmaze-201
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| |
| . |
| = S |
| . O . . |
| + o * + . |
| . + %.+ |
| E+X. |
+-----------------+
Then enter the "ssh-copy-id" command on the intsmaze-201 node to copy and append the public key to the authorization list file authorized_keys of intsmaze-201 and intsmaze-202.
[intsmaze@intsmaze-201 ~]$ ssh-copy-id intsmaze-201
[intsmaze@intsmaze-201 ~]$ ssh-copy-id intsmaze-202
Repeat the operation on the intsmaze-201 node on the intsmaze-202 node, and then configure the password-free login.
2 Standalone cluster
After the secret-free login is configured, first create a folder named standalone under the /home/intsmaze/flink/ path on the intsmaze-201 and intsmaze-202 nodes. Upload the flink-XX.tgz package to the intsmaze-201 node, and use the tar command to decompress it to the /home/intsmaze/flink/standalone/ folder.
After everything is ready, enter the <flink-home> directory to configure it. The main configuration files involved here are <flink-home>/conf/flink-conf.yam and <flink-home>/conf/slaves files.
1. flink-conf.yaml file configuration
The following list suggests modifying the configuration values in the <flink-home>/conf/flink-conf.yaml file, which indicates that a JobManager will be run on the intsmaze-201 node.
[intsmaze@intsmaze-201 conf]$ vim <flink-home>/conf/flink-conf.yaml
# 必选，将jobmanager.rpc.address的值设置成Flink集群中Jobmanager节点的IP地址
jobmanager.rpc.address: intsmaze-201
# 可选，默认每个TaskManager的任务槽为1
# 值越大表示可供运行的Flink作业越多，一般根据每台机器的可用cpu的数量进行设置。
taskmanager.numberOfTaskSlots: 6
# 可选: 每个JobManager可用内存的大小，内存参数的单位是MB
jobmanager.heap.mb: 1024
# 可选：每个TaskManager可用内存的大小，内存参数的单位是MB
taskmanager.heap.mb: 1024
2. slaves file configuration
A list of TaskManager nodes must be provided for the Flink cluster. You need to edit the <flink-home>/conf/slaves file and add the IP (or hostname) of each TaskManager node in the file.
Next, add the host names intsmaze-201 and intsmaze-202 to the slaves file to indicate that a TaskManager will run on each of the two nodes.
[intsmaze@intsmaze-201 conf]$ vim <flink-home>/conf/slaves
intsmaze-201
intsmaze-202
3. Distributing the Flink directory
After configuring the parameters of Flink's standalone mode under <flink-home>/conf of the intsmaze-201 node, you need to ensure that the files in the <flink-home> directory must be available in the same path on each TaskManager. For this purpose, use the scp command to copy the entire <flink-home> directory to each working node.
[intsmaze@intsmaze-201 conf]$ scp -r /home/intsmaze/flink/standalone/flink-1.7.2 intsmaze-202:/home/intsmaze/flink/standalone/
4. Start a Flink cluster
After distributing the <flink-home> directory, start the Flink cluster on the intsmaze-201 node through the /bin/start-cluster.sh script.
[intsmaze@intsmaze-201 flink-1.7.2]$ ./bin/start-cluster.sh
After the startup is successful, the job manager process will start a Flink cluster WEB console page on port 8081 by default. You can enter http://intsmaze-201:8081 in the browser to view the WEB page of the job manager. If everything is running normally, two available task manager instances will be displayed on the WEB page, and the number of task slots in the cluster is 12.
In the <flink-home>/bin/ directory, you can use the following script to shut down the started Flink cluster.
[intsmaze@intsmaze-201 flink-1.7.2]$ ./bin/stop-cluster.sh
5. Cluster node restart and expansion
Flink also provides scripts to dynamically expand cluster nodes or restart specified nodes in the cluster. You can run the <flink-home>/bin/jobmanager.sh and <flink-home>/bin/taskmanager.sh scripts.
Restart or expand the JobManager node
bin/jobmanager.sh ((start|start-foreground) cluster)|stop|stop-all
Restart or expand the TaskManager node
bin/taskmanager.sh start|start-foreground|stop|stop-all