9090监控页面显示storage都没用(这里似乎只统计persist持久化的):
但是jmap -heap pid显示内存不够了:
然后报错:SparkException: Kryo serialization failed: Buffer overflow
解决办法:加大spark.driver.memory、spark.executor.memory、spark.kryoserializer.buffer
或者在 submit时候配置:
--conf "spark.kryoserializer.buffer.max=384"
--driver-memory 5g
--conf spark.executor.memory=6g
--total-executor-cores 12
--conf spark.executor.cores=3