2. Configure the history server.
Add the following configuration in spark-3.1.1-bin-hadoop2.7/conf/spark-defaults.conf, where d:/log/spark is the log storage location
spark.eventLog.enabled true
spark.eventLog. dir file:///d:/log/spark
spark.eventLog.compress true
spark.history.fs.logDirectory file:///d:/log/spark
Use the command to start the history server. You can see the started port number. Access the history server
Spark-class org.apache.spark.deploy.history.HistoryServer through http://localhost:port number
Add the following configuration .config("spark.eventLog.enabled", value = true)
.config("spark.eventLog.dir", "file:///D:/log/spark") in SparkSession in scala code
. config(“spark.history.fs.logDirectory”, “file:///D:/log/spark”)