PySpark与jupyer notebook

环境主备及环境配置：

JAVA_HOME=/root/jdk1.8.0_144

JAVA_BIN=/root/jdk1.8.0_144/bin

JRE_HOME=/root/jdk1.8.0_144/jre

CLASSPATH=/root/jdk1.8.0_144/jre/lib:/root/jdk1.8.0_144/lib:/root/jdk1.8.0_144/jre/lib/charsets.jar

SCALA_HOME=/root/scala-2.12.0

SPARK_HOME=/root/spark-2.4.4

export PYSPARK_PYTHON=/root/anaconda3/bin/python3

export PYSPARK_DRIVER_PYTHON=jupyter

export PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:/root/anaconda3/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$PATH

日志级别修改

mv log4j.properties.template log4j.properties

/root/spark-2.4.4/conf/log4j.properties

log4j.rootCategory=ERROR, console

(base) [root@pyspark bin]# run-example SparkPi 10
19/10/22 22:08:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pi is roughly 3.143691143691144
(base) [root@pyspark bin]# run-example SparkPi 20
19/10/22 22:12:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pi is roughly 3.1413995706997855
(base) [root@pyspark bin]# run-example SparkPi 110
19/10/22 22:12:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pi is roughly 3.141459921950902