PyCharm remote connection (linux) python interpreter imports pyspark and reports an error: No module named 'pyspark' , and when from pyspark import SparkConf, SparkContext and other packages, the following occurs when there is a red wavy line under pyspark
Linux can import pyspark package normally, but PyCharm can't
My solution is to decompress the compressed packages of 'py4j' and 'pyspark' in the python folder in the spark installation directory to lib/python3.10/site-packages/ of the python interpreter or decompress to this folder to create a soft link can also
The following are the specific operations:
1. Enter the spark installation directory
cd /opt/module/spark/python/lib/
ll
#自己spark的安装路径下的python文件夹下的lib文件夹You will see the compressed packages of ' py4j-0.10.9.5-src.zip ' and ' pyspark.zip '
2. Unzip to lib/python3.10/site-packages under the python interpreter
tar -zxvf pyspark.zip -C /usr/local/python3/lib/python3.10/site-packages/
#/usr/local/python3是我自己的python解释器路径
tar -zxvf py4j-0.10.9.5-src.zip.zip -C /usr/local/python3/lib/python3.10/site-packages/
#/usr/local/python3是我自己的python解释器路径cd /usr/local/python3/lib/python3.10/site-packages/ can be seen
3. If you can’t decompress , you can drag the two compressed packages to the windows computer after decompression, and then drag them into the corresponding path of the python interpreter of the virtual machine
Run the test code: (Although the import package is still popular, at least the spark code can be run)
It should be red when there is no corresponding info document (no pyspark and py4j version information) when importing