java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
在stackoverflow.com上找到的解决办法
修改spark/python/pyspark目录下worker.py文件
修改 def process():为以下内容
def process(): iterator = deserializer.load_stream(infile) serializer.dump_stream(func(split_index, iterator), outfile) for obj in iterator: pass
然后将pyspark目录打包为zip 替换掉spark/python/lib/目录下对应的pyspark.zip文件