Warning: Master endpoint spark://node01:7077 was not a REST server. Falling back to legacy submission gateway instead.
在standalone集群模式下,有两个提交网关:
1.使用org.apache.spark.deploy.Client作为包装器来使用传统的RPC网关
2.Spark 1.3中引入的基于rest的网关
第二种方法是Spark 1.3的默认行为,但是Spark submit将会失败
如果master不是一个REST服务器,那么它将无法使用REST网关。
if (args.isStandaloneCluster && args.useRest) {
try {
printStream.println("Running Spark using the REST application submission protocol.")
// 调用doRunMain()方法
doRunMain()
} catch {
// Fail over to use the legacy submission gateway
case e: SubmitRestConnectionException =>
printWarning(s"Master endpoint ${args.master} was not a REST server. " +
"Falling back to legacy submission gateway instead.")
args.useRest = false
submit(args)
}
// 其他模式,直接调用doRunMain方法
} else {
doRunMain()
}