Python Spark实现协同过滤算法

Python Spark实现协同过滤算法

环境

Python 3.7.6
CentOS Linux release 8.1.1911 (Core) 
Spark version 3.0.0
Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 14.0.2)

实现代码

from pyspark import SparkContext

# $example on$
from pyspark.mllib.recommendation import ALS, Rating
from argparse import ArgumentParser

if __name__ == "__main__":
    sc = SparkContext(appName="PythonCollaborativeFilteringExample")
    # dataset path
    data = sc.textFile('./test.data')
    
    # RDD of Rating or (userID, productID, rating) tuple.
    # Rating(user=l[0], product=l[1], rating=l[2])
    ratings = data.map(lambda l: l.split(','))\
        .map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))

	# Number of features to use (also referred to as the number of latent factors).
    rank = 10
    # Number of iterations of ALS. (default: 5)
    numIterations = 10
    model = ALS.train(ratings, rank, numIterations)

    testdata = ratings.map(lambda p: (p[0], p[1]))
    predictions = model.predictAll(testdata).map(lambda r: ((r[0], r[1]), r[2]))
    ratesAndPreds = ratings.map(lambda r: ((r[0], r[1]), r[2])).join(predictions)
    MSE = ratesAndPreds.map(lambda r: (r[1][0] - r[1][1])**2).mean()
    print("Mean Squared Error = " + str(MSE))

	# (userId, productID)
    print(model.predict(4, 4))

命令行

python recommendation_example.py

测试数据

1,1,5.0
1,2,1.0
1,3,5.0
1,4,1.0
2,1,5.0
2,2,1.0
2,3,5.0
2,4,1.0
3,1,1.0
3,2,5.0
3,3,1.0
3,4,5.0
4,1,1.0
4,2,5.0
4,3,1.0
4,4,5.0

引用

周志华. 机器学习 : = Machine learning[M]. 清华大学出版社, 2016.
[美] 伊恩·古德费洛 / [加] 约书亚·本吉奥 / [加] 亚伦·库维尔. 深度学习. 人民邮电出版社, 2017.
HoldenKarau. Spark快速大数据分析[M]. 人民邮电出版社, 2015.
Apache Spark Docs
Apache Spark API

最后

• 由于博主水平有限，不免有疏漏之处，欢迎读者随时批评指正，以免造成不必要的误解！