pyspark java.net.SocketException: Connection reset by peer

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/qq_31772441/article/details/86560437

在window、运行pyspark训练模型，报错
Caused by: java.net.SocketException: Connection reset by peer: socket write error

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.net.SocketException: Connection reset by peer: socket write error
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
	...
Caused by: java.net.SocketException: Connection reset by peer: socket write error
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
	...

解决：修改python环境下的pyspark包中的worker. py 中的process函数

（注：python环境下的pyspark包是从spark目录拷贝过来的）
eg: E:\Anaconda3\envs\py27\Lib\site-packages\pyspark\worker. py
定位到 process 函数：

        def process():
            iterator = deserializer.load_stream(infile)
            serializer.dump_stream(func(split_index, iterator), outfile)
            # 添加以下内容
            for obj in interator:
                pass

具体原因待探究…

---

end