Spark movie recommendation system based on (recommended system 1-7)
22/100
publish articles
liuge36
Part IV - Recommended system - real-time recommendations
This module model in Section 4 was based, began to make real-time recommendations for the user, the user is most likely to recommend a favorite five movie.
A couple of points
1. Source data testData test set. This is inside the user, may be present in the training set, it could be a new user. So, here to do the processing.
- SparkStreaming + kakfa
Coding Start ##
Step a: In streaming packet, new PopularMovies2
package com.csylh.recommend.streaming
import com.csylh.recommend.config.AppConf
import org.apache.spark.sql.SaveMode
/**
* Description: 个性化推荐
*
* @Author: 留歌36
* @Date: 2019/10/18 17:42
*/
object PopularMovies2 extends AppConf{
def main(args: Array[String]): Unit = {
val movieRatingCount = spark.sql("select count(*) c, movieid from trainingdata group by movieid order by c")
// 前5部进行推荐
val Top5Movies = movieRatingCount.limit(5)
Top5Movies.registerTempTable("top5")
val top5DF = spark.sql("select a.title from movies a join top5 b on a.movieid=b.movieid")
// 把数据写入到HDFS上
top5DF.write.mode(SaveMode.Overwrite).parquet("/tmp/top5DF")
// 将数据从HDFS加载到Hive数据仓库中去
spark.sql("drop table if exists top5DF")
spark.sql("create table if not exists top5DF(title string) stored as parquet")
spark.sql("load data inpath '/tmp/top5DF' overwrite into table top5DF")
// 最终表里应该是5部推荐电影的名称
}
}
Step Two: In streaming packet, new SparkDirectStreamApp
package com.csylh.recommend.streaming
import com.csylh.recommend.config.AppConf
import kafka.serializer.StringDecoder
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
/**
* Description:
*
* @Author: 留歌36
* @Date: 2019/10/18 16:33
*/
object SparkDirectStreamApp extends AppConf{
def main(args:Array[String]): Unit ={
val ssc = new StreamingContext(sc, Seconds(5))
val topics = "movie_topic".split(",").toSet
val kafkaParams = Map[String, String](
"metadata.broker.list"->"hadoop001:9093,hadoop001:9094,hadoop001:9095",
"auto.offset.reset" -> "largest" //smallest :从头开始 largest:最新
)
// Direct 模式:SparkStreaming 主动去Kafka中pull拉数据
val modelPath = "/tmp/BestModel/0.8521581387523667"
val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)
def exist(u: Int): Boolean = {
val trainingdataUserIdList = spark.sql("select distinct(userid) from trainingdata")
.rdd
.map(x => x.getInt(0))
.collect() // RDD[row] ==> RDD[Int]
trainingdataUserIdList.contains(u)
}
// 为没有登录的用户推荐电影的策略:
// 1.推荐观看人数较多的电影,采用这种策略
// 2.推荐最新的电影
val defaultrecresult = spark.sql("select * from top5DF").rdd.toLocalIterator
// 创建SparkStreaming接收kafka消息队列数据的2种方式
// 一种是Direct approache,通过SparkStreaming自己主动去Kafka消息队
// 列中查询还没有接收进来的数据,并把他们拉pull到sparkstreaming中。
val model = MatrixFactorizationModel.load(ssc.sparkContext, modelPath)
val messages = stream.foreachRDD(rdd=> {
val userIdStreamRdd = rdd.map(_._2.split("|")).map(x=>x(1)).map(_.toInt)
val validusers = userIdStreamRdd.filter(userId => exist(userId))
val newusers = userIdStreamRdd.filter(userId => !exist(userId))
// 采用迭代器的方式来避开对象不能序列化的问题。
// 通过对RDD中的每个元素实时产生推荐结果,将结果写入到redis,或者其他高速缓存中,来达到一定的实时性。
// 2个流的处理分成2个sparkstreaming的应用来处理。
val validusersIter = validusers.toLocalIterator
val newusersIter = newusers.toLocalIterator
while (validusersIter.hasNext) {
val u= validusersIter.next
println("userId"+u)
val recresult = model.recommendProducts(u, 5)
val recmoviesid = recresult.map(_.product)
println("我为用户" + u + "【实时】推荐了以下5部电影:")
for (i <- recmoviesid) {
val moviename = spark.sql(s"select title from movies where movieId=$i").first().getString(0)
println(moviename)
}
}
while (newusersIter.hasNext) {
println("*新用户你好*以下电影为您推荐below movies are recommended for you :")
for (i <- defaultrecresult) {
println(i.getString(0))
}
}
})
ssc.start()
ssc.awaitTermination()
}
}
Step three: the project will be created to package uploaded to the server
mvn clean package -Dmaven.test.skip=true
Step four: write your personalized referral code execution shell script
[root@hadoop001 ml]# vim PopularMovies2.sh
export HADOOP_CONF_DIR=/root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
$SPARK_HOME/bin/spark-submit \
--class com.csylh.recommend.streaming.PopularMovies2 \
--master spark://hadoop001:7077 \
--name PopularMovies2 \
--driver-memory 10g \
--executor-memory 5g \
/root/data/ml/movie-recommend-1.0.jar
Step five: Perform sh PopularMovies2.sh
make sure:
[root@hadoop001 ml]# spark-sql
19/10/20 22:59:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1571583574311
spark-sql> show tables;
default links false
default movies false
default ratings false
default tags false
default testdata false
default top5df false
default trainingdata false
default trainingdataasc false
default trainingdatadesc false
Time taken: 2.232 seconds, Fetched 9 row(s)
spark-sql> select * from top5df;
Follow the Bitch (1996)
Radio Inside (1994)
Faces of Schlock (2005)
Mág (1988)
"Son of Monte Cristo
Time taken: 1.8 seconds, Fetched 5 row(s)
spark-sql>
Step six: Re-write model real-referral code execution shell script
export HADOOP_CONF_DIR=/root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
$SPARK_HOME/bin/spark-submit \
--class com.csylh.recommend.streaming.SparkDirectStreamApp \
--master spark://hadoop001:7077 \
--name SparkDirectStreamApp \
--driver-memory 10g \
--executor-memory 5g \
--total-executor-cores 10 \
--jars /root/app/kafka_2.11-1.1.1/libs/kafka-clients-1.1.1.jar \
--packages "mysql:mysql-connector-java:5.1.38,org.apache.spark:spark-streaming-kafka-0-8_2.11:2.4.2" \
/root/data/ml/movie-recommend-1.0.jar
Step seven: sh SparkDirectStreamApp.sh
// ALL...
Have any questions, please leave a message with the exchange ~~
More Articles: Spark movie recommendation system based on: https: //blog.csdn.net/liuge36/column/info/29285
Part IV - Recommended system - real-time recommendations
of this module model in Section 4 was based, began to make real-time recommendations for the user, the user is most likely to recommend a favorite five movie.
Some instructions
1. The data source is the data testData test set. This is inside the user, may be present in the training set, it could be a new user. So, here to do the processing.
- SparkStreaming + kakfa
here Insert Picture Description
Coding Start
Step a: In streaming packet, new PopularMovies2
package com.csylh.recommend.streaming
import com.csylh.recommend.config.AppConf
import org.apache.spark.sql.SaveMode
/**
- Description: Personalized Recommendation
- @Author: stay 36 songs
@Date: 2019/10/18 17:42
/
object PopularMovies2 extends AppConf{
def main(args: Array[String]): Unit = {
val movieRatingCount = spark.sql("select count() c, movieid from trainingdata group by movieid order by c")
// 前5部进行推荐
val Top5Movies = movieRatingCount.limit(5)Top5Movies.registerTempTable("top5") val top5DF = spark.sql("select a.title from movies a join top5 b on a.movieid=b.movieid") // 把数据写入到HDFS上 top5DF.write.mode(SaveMode.Overwrite).parquet("/tmp/top5DF") // 将数据从HDFS加载到Hive数据仓库中去 spark.sql("drop table if exists top5DF") spark.sql("create table if not exists top5DF(title string) stored as parquet") spark.sql("load data inpath '/tmp/top5DF' overwrite into table top5DF") // 最终表里应该是5部推荐电影的名称
}
}
Step Two: In streaming packet, new SparkDirectStreamApp
package com.csylh.recommend.streaming
import com.csylh.recommend.config.AppConf
import kafka.serializer.StringDecoder
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
/**
- Description:
- @Author: stay 36 songs
@Date: 2019/10/18 16:33
*/
object SparkDirectStreamApp extends AppConf{
def main(args:Array[String]): Unit ={
val ssc = new StreamingContext(sc, Seconds(5))val topics = "movie_topic".split(",").toSet
kafkaParams = the Map Val [String, String] (
"metadata.broker.list" -> "hadoop001: 9093, hadoop001: 9094, hadoop001: 9095",
"auto.offset.reset" -> "Largest" // Smallest: from scratch began to largest: the latest
)
// Direct mode: SparkStreaming the initiative to pull pull data Kafka in
Val the modelPath = "/tmp/BestModel/0.8521581387523667"
Val Stream = KafkaUtils.createDirectStream String, String, StringDecoder, StringDecoderdef exist(u: Int): Boolean = {
val trainingdataUserIdList = spark.sql("select distinct(userid) from trainingdata")
.rdd
.map(x => x.getInt(0))
.collect() // RDD[row] ==> RDD[Int]trainingdataUserIdList.contains(u)
}
// movies recommended for users not logged strategy:
// 1. Recommended higher number of movies to watch, this strategy
// 2. Recommended latest movies
val defaultrecresult = spark.sql ( "select * from top5DF") .rdd.toLocalIterator// Create SparkStreaming receiving kafka message queue data in two ways
// One is Direct approache, through their own initiative to Kafka SparkStreaming team news
// column inquiry has not yet received incoming data, and bring them to pull in sparkstreaming.val model = MatrixFactorizationModel.load(ssc.sparkContext, modelPath)
val messages = stream.foreachRDD(rdd=> {val userIdStreamRdd = rdd.map(_._2.split("|")).map(x=>x(1)).map(_.toInt) val validusers = userIdStreamRdd.filter(userId => exist(userId)) val newusers = userIdStreamRdd.filter(userId => !exist(userId)) // 采用迭代器的方式来避开对象不能序列化的问题。 // 通过对RDD中的每个元素实时产生推荐结果,将结果写入到redis,或者其他高速缓存中,来达到一定的实时性。 // 2个流的处理分成2个sparkstreaming的应用来处理。 val validusersIter = validusers.toLocalIterator val newusersIter = newusers.toLocalIterator while (validusersIter.hasNext) { val u= validusersIter.next println("userId"+u) val recresult = model.recommendProducts(u, 5) val recmoviesid = recresult.map(_.product) println("我为用户" + u + "【实时】推荐了以下5部电影:") for (i <- recmoviesid) { val moviename = spark.sql(s"select title from movies where movieId=$i").first().getString(0) println(moviename) } } while (newusersIter.hasNext) { println("*新用户你好*以下电影为您推荐below movies are recommended for you :") for (i <- defaultrecresult) { println(i.getString(0)) } }
})
ssc.start()
ssc.awaitTermination()
}
}
Step three: the project will be created to package uploaded to the server
mvn clean package -Dmaven.test.skip = true
Step four: write your personalized referral code execution shell script
[root@hadoop001 ml]# vim PopularMovies2.sh
export HADOOP_CONF_DIR=/root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
$SPARK_HOME/bin/spark-submit --class com.csylh.recommend.streaming.PopularMovies2 --master spark://hadoop001:7077 --name PopularMovies2 --driver-memory 10g --executor-memory 5g /root/data/ml/movie-recommend-1.0.jar
Step five: Perform sh PopularMovies2.sh
make sure:
[root@hadoop001 ml]# spark-sql
19/10/20 22:59:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1571583574311
spark-sql> show tables;
default links false
default movies false
default ratings false
default tags false
default testdata false
default top5df false
default trainingdata false
default trainingdataasc false
default trainingdatadesc false
Time taken: 2.232 seconds, Fetched 9 row(s)
spark-sql> select * from top5df;
Follow the Bitch (1996)
Radio Inside (1994)
Faces of Schlock (2005)
Mág (1988)
"Son of Monte Cristo
Time taken: 1.8 seconds, Fetched 5 row(s)
spark-sql>
Step six: Re-write model real-referral code execution shell script
export HADOOP_CONF_DIR=/root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
$SPARK_HOME/bin/spark-submit --class com.csylh.recommend.streaming.SparkDirectStreamApp --master spark://hadoop001:7077 --name SparkDirectStreamApp --driver-memory 10g --executor-memory 5g --total-executor-cores 10 --jars /root/app/kafka_2.11-1.1.1/libs/kafka-clients-1.1.1.jar --packages "mysql:mysql-connector-java:5.1.38,org.apache.spark:spark-streaming-kafka-0-8_2.11:2.4.2" /root/data/ml/movie-recommend-1.0.jar
步骤七:sh SparkDirectStreamApp.sh
// ALL…
Have any questions, please leave a message with the exchange ~~
More Articles: Spark movie recommendation system based on: https: //blog.csdn.net/liuge36/column/info/29285
The number 214 of words in the current row Markdown 5232 rows 1, 0HTML 4908 words of paragraph 154 of the current column
Translation