连接部分代码
from pyspark import SparkContext, SQLContext, SparkConf
from pyspark.sql import SparkSession
import time, sys, warnings
MONGOURI = 'mongodb://172.00.100.11:27017/{}.{}'.format('ad_portrayal', 'user_portrayal_test')
sc = SparkContext(appName="deal_with_portrayal_data")
sqlContext = SQLContext(sc)
ss = SparkSession(sc).builder \
.config("spark.mongodb.input.uri", MONGOURI) \
.config("spark.mongodb.output.uri", MONGOURI) \
.getOrCreate()
warnings.simplefilter(action='ignore')
sc.setLogLevel('ERROR')
spark提交代码
- packages指定驱动,如下,2.11为Scala版本,2.4.3为spark版本。版本号一定要和集群的对应
PYSPARK_PYTHON=/data/anaconda3/bin/python3 \
/opt/spark/bin/spark-submit \
--master yarn \
--deploy-mode client \
--driver-memory 4g \
--executor-memory 4g \
--driver-cores 4 \
--executor-cores 4 \
--packages org.mongodb.spark:mongo-spark-connector_2.11:2.4.3 \
--name 'deal_with_portrayal_data' \
deal_portData.py >>./log/deal_portData.log 2>>./log/deal_portData.err
然和可以读数据测试了
coll = ss.read.format('mongo').load()
coll.show()
print(type(coll))
+-----+---------+-------+
|EVENT| REQID| _id|
+-----+---------+-------+
|click|DWADLAWMD|ABCDEFG|
+-----+---------+-------+
<class 'pyspark.sql.dataframe.DataFrame'>