Spark-MLlib的快速使用之二（朴素贝叶斯分类）

（1）算法描述

算法介绍：

朴素贝叶斯法是基于贝叶斯定理与特征条件独立假设的分类方法。

朴素贝叶斯的思想基础是这样的：对于给出的待分类项，求解在此项出现的条件下各个类别出现的概率，在没有其它可用信息下，我们会选择条件概率最大的类别作为此待分类项应属的类别。

（2）测试数据

1 125:145 126:255 127:211 128:31 152:32 153:237 154:253 155:252 156:71 180:11 181:175 182:253 183:252 184:71 209:144 210:253 211:252 212:71 236:16 237:191 238:253 239:252 240:71 264:26 265:221 266:253 267:252 268:124 269:31 293:125 294:253 295:252 296:252 297:108 322:253 323:252 324:252 325:108 350:255 351:253 352:253 353:108 378:253 379:252 380:252 381:108 406:253 407:252 408:252 409:108 434:253 435:252 436:252 437:108 462:255 463:253 464:253 465:170 490:253 491:252 492:252 493:252 494:42 518:149 519:252 520:252 521:252 522:144 546:109 547:252 548:252 549:252 550:144 575:218 576:253 577:253 578:255 579:35 603:175 604:252 605:252 606:253 607:35 631:73 632:252 633:252 634:253 635:35 659:31 660:211 661:252 662:253 663:35

（3）测试程序

public class JavaNaiveBayesExample {

public static void main(String[] args) {

SparkConf sparkConf = new SparkConf().setAppName("JavaNaiveBayesExample").setMaster("local");

扫描二维码关注公众号，回复： 4139319 查看本文章

JavaSparkContext jsc = new JavaSparkContext(sparkConf);

// $example on$

String path = "sample_libsvm_data.txt";

JavaRDD<LabeledPoint> inputData = MLUtils.loadLibSVMFile(jsc.sc(), path).toJavaRDD();

JavaRDD<LabeledPoint>[] tmp = inputData.randomSplit(new double[]{0.6, 0.4}, 12345);

JavaRDD<LabeledPoint> training = tmp[0]; // training set

JavaRDD<LabeledPoint> test = tmp[1]; // test set

final NaiveBayesModel model = NaiveBayes.train(training.rdd(), 1.0);

JavaPairRDD<Double, Double> predictionAndLabel =

test.mapToPair(new PairFunction<LabeledPoint, Double, Double>() {

@Override

public Tuple2<Double, Double> call(LabeledPoint p) {

return new Tuple2<Double, Double>(model.predict(p.features()), p.label());

}

});

System.out.println("----------------->"+predictionAndLabel.take(10));

double accuracy = predictionAndLabel.filter(new Function<Tuple2<Double, Double>, Boolean>() {

@Override

public Boolean call(Tuple2<Double, Double> pl) {

return pl._1().equals(pl._2());

}

}).count() / (double) test.count();

// Save and load model

model.save(jsc.sc(), "target/tmp/myNaiveBayesModel");

NaiveBayesModel sameModel = NaiveBayesModel.load(jsc.sc(), "target/tmp/myNaiveBayesModel");

// $example off$

}

}

（4）测试结果

[(1.0,1.0), (0.0,0.0), (1.0,1.0), (0.0,0.0), (0.0,0.0), (1.0,1.0), (0.0,0.0), (0.0,0.0), (0.0,0.0), (0.0,0.0)]