Spark implements KMeans algorithm code example

Spark implements K-Means algorithm code example - K-Means algorithm is a distance-based clustering algorithm that uses an iterative method to calculate K cluster centers and cluster several points into K categories. MLlib implements the principle of K-Means algorithm

The K-Means algorithm is a distance-based clustering algorithm that uses an iterative method to calculate K cluster centers and cluster several points into K categories.

The principle of MLlib implementing the K-Means algorithm is to run multiple K-Means algorithms, each called a run, to return the cluster center of the best cluster. The initial cluster center can be random or obtained from KMean||. The algorithm ends when the iteration reaches a certain number of times or when all runs converge.

Use Spark to implement the K-Means algorithm. First modify the pom file and introduce the machine learning MLlib package:


   org.apache.spark
   spark-mllib_2.10
   1.6.0

Code:

import org.apache.log4j.{Level,Logger}
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.mllib.clustering.KMeans
import org.apache.spark.mllib.linalg.Vectors
object Kmeans {
 def main(args:Array[String]) = {
 // 屏蔽日志
 Logger.getLogger("org.apache.spark").setLevel(Level.WARN)
 Logger.getLogger("org.apache.jetty.server").setLevel(Level.OFF)
 // 设置运行环境
 val cOnf= new SparkConf().setAppName("K-Means").setMaster("spark://master:7077")
  .setJars(Seq("E:\\Intellij\\Projects\\SimpleGraphX\\SimpleGraphX.jar"))
 val sc = new SparkContext(conf)
 // 装载数据集
 val data = sc.textFile("hdfs://master:9000/kmeans_data.txt", 1)
 val parsedData = data.map(s => Vectors.dense(s.split(" ").map(_.toDouble)))
 // 将数据集聚类,2个类,20次迭代,形成数据模型
 val numClusters = 2
 val numIteratiOns= 20
 val model = KMeans.train(parsedData, numClusters, numIterations)
 // 数据模型的中心点
 println("Cluster centres:")
 for(c <- model.clusterCenters) {
  println(" " + c.toString)
 }
 // 使用误差平方之和来评估数据模型
 val cost = model.computeCost(parsedData)
 println("Within Set Sum of Squared Errors = " + cost)
 // 使用模型测试单点数据
 println("Vectors 7.3 1.5 10.9 is belong to cluster:" + model.predict(Vectors.dense("7.3 1.5 10.9".split(" ")
  .map(_.toDouble))))
 println("Vectors 4.2 11.2 2.7 is belong to cluster:" + model.predict(Vectors.dense("4.2 11.2 2.7".split(" ")
  .map(_.toDouble))))
 println("Vectors 18.0 4.5 3.8 is belong to cluster:" + model.predict(Vectors.dense("1.0 14.5 73.8".split(" ")
  .map(_.toDouble))))
 // 返回数据集和结果
 val result = data.map {
  line =>
  val linevectore = Vectors.dense(line.split(" ").map(_.toDouble))
  val prediction = model.predict(linevectore)
  line + " " + prediction
 }.collect.foreach(println)
 sc.stop
 }
}

Use the textFile() method to load the data set and obtain the RDD, and then use the KMeans.train() method to obtain a KMeans model based on the RDD, K value and number of iterations. After obtaining the KMeans model, you can determine which class a set of data belongs to. The specific method is to use the Vectors.dense() method to generate a Vector, and then use the KMeans.predict() method to return which class it belongs to.

operation result:

Cluster centres:
 [6.062499999999999,6.7124999999999995,11.5]
 [3.5,12.2,60.0]
Within Set Sum of Squared Errors = 943.2074999999998
Vectors 7.3 1.5 10.9 is belong to cluster:0
Vectors 4.2 11.2 2.7 is belong to cluster:0
Vectors 18.0 4.5 3.8 is belong to cluster:1
0.0 0.0 5.0 0
0.1 10.1 0.1 0
1.2 5.2 13.5 0
9.5 9.0 9.0 0
9.1 9.1 9.1 0
19.2 9.4 29.2 0
5.8 3.0 18.0 0
3.5 12.2 60.0 1
3.6 7.9 8.1 0

Spark implements KMeans algorithm code example

Guess you like