spark参数调整

直接报错  spark Kryo serialization failed: Buffer overflow 错误

Caused by: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 28. To avoid this, increase spark.kryoserializer.buffer.max value

提示需要调整的参数是 spark.kryoserializer.buffer.max 最少是28  

直接在 submit时候配置算了

 --conf 'spark.kryoserializer.buffer.max=64'

有时候设置 

sparkConf.set("spark.kryoserializer.buffer","64");