Spark Submit the jar package to the cluster to run an exception java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster

java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster

Problem Description:

Spark submits the jar package to the cluster and reports an exception

java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster

---

Cause Analysis:

Spark cannot find information about the hadoop cluster

---

solution:

Add the Hadoop configuration file path to the spark-env.sh configuration file

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop  

Restart spark