安装环境信息
flink-1.6.2-bin-hadoop27-scala_2.11.tgz
hadoop-2.7.5
java 1.8
zookeeper 3.4.6
os:centos 6.4
1、下载
直接去flink的社区下载就可以了。http://flink.apache.org/downloads.html
2、解压
tar -zxvf flink-1.6.2-bin-hadoop27-scala_2.11.tgz
1
3、修改环境变量 ~.bash_profile
export FLINK_HOME=/opt/flink-1.6.2
export PATH=$FLINK_HOME/bin:$PATH
4、修改flink-conf.yaml配置文件，先配置一个简单版本，standalone的模式
Hadoop的nameservice
jobmanager.rpc.address: cdh1
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.heap.size: 1024m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 12
5、修改slaves和masters2个文件，用来配置taskManager和JobManager信息
[hadoop@cdh1 conf]$ cat slaves
cdh2
cdh3
cdh4
cdh5
[hadoop@cdh1 conf]$ cat masters
cdh1:8081
6、将flink安装所有信息已经环境信息同步到其他机器上面，这里有几台机器就要执行几次
scp .bash_profile hadoop@cdh3:~/.bash_profile
scp -r ./flink-1.6.2 hadoop@cdh3:/opt/
7、启动flink
[hadoop@cdh1 bin]$ ./start-cluster.sh
8、启动完成已经我们可以用jps。分别可以看到JobManager和TaskManager的2个进程
[hadoop@cdh1 bin]$ jps
3866 StandaloneSessionClusterEntrypoint
[hadoop@cdh2 ~]$ jps
3534 TaskManagerRunner
8、登录JobManager的地址查看ui http://192.168.18.160:8081
已经表示搭建完成了，现在我们开始验证一下集群
使用start-scala-shell.sh来验证
${FLINK_HOME}/bin/start-scala-shell.sh是flink提供的交互式clinet,可以用于代码片段的测试，方便开发工作，它有两种启动方式，一种是工作在本地，另一种是工作到集群。本例中因为机器连接非常方便，就直接使用集群进行测试，在开发中，如果集群连接不是非常方便，可以连接到本地，在本地开发测试通过后，再连接到集群进行部署工作。如果程序有依赖的jar包，则可以使用 -a <path/to/jar.jar> 或 --addclasspath <path/to/jar.jar>参数来添加依赖。
1.本地连接
${FLINK_HOME}/bin/start-scala-shell.sh local
1
2.集群连接
${FLINK_HOME}/bin/start-scala-shell.sh remote <hostname> <portnumber>
1
3.带有依赖包的格式
${FLINK_HOME}/bin/start-scala-shell.sh [local|remote<host><port>] --addclasspath<path/to/jar.jar>
1
4.查看帮助
${FLINK_HOME}/bin/start-scala-shell.sh --help
[hadoop@cdh2 bin]$ ./start-scala-shell.sh --help
Flink Scala Shell
Usage: start-scala-shell.sh [local|remote|yarn] [options] <args>...
Command: local [options]
Starts Flink scala shell with a local Flink cluster
-a, --addclasspath <path/to/jar>
Specifies additional jars to be used in Flink
Command: remote [options] <host> <port>
Starts Flink scala shell connecting to a remote cluster
<host> Remote host name as string
<port> Remote port as integer
-a, --addclasspath <path/to/jar>
Specifies additional jars to be used in Flink
Command: yarn [options]
Starts Flink scala shell connecting to a yarn cluster
-n, --container arg Number of YARN container to allocate (= Number of TaskManagers)
-jm, --jobManagerMemory arg
Memory for JobManager container
-nm, --name <value> Set a custom name for the application on YARN
-qu, --queue <arg> Specifies YARN queue
-s, --slots <arg> Number of slots per TaskManager
-tm, --taskManagerMemory <arg>
Memory per TaskManager container
-a, --addclasspath <path/to/jar>
Specifies additional jars to be used in Flink
--configDir <value> The configuration directory.
-h, --help Prints this usage text
我们 使用集群模式去验证
[hadoop@cdh1 bin]$ ./start-scala-shell.sh remote 192.168.18.160 8081
1
运行如下案例代码
Scala> val text = benv.fromElements(
"To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer",
"The slings and arrows of outrageous fortune",
"Or to take arms against a sea of troubles,")
Scala> val counts = text
.flatMap { _.toLowerCase.split("\\W+") }
.map { (_, 1) }.groupBy(0).sum(1)
Scala> counts.print()
运行结果
---------------------
