Scala总结
凡事豫则立，不豫则废；言前定，则不跲；事前定，则不困。 — 《礼记·中庸》
官网：http://www.scala-lang.org/download/
概述
听说scala是一门神一样的语言，接下来咱们也来了解一下这门神奇的语言。
文件类型
- 后缀为class的：相当于创建一个类
- 后缀为Object的：相当于创建一个对象，或者叫做伴生对象
- 后缀为Trait的：相当于接口
环境安装
1-1）、安装JDK
运行scala需要JDK的环境，所以在安装scala之前需要安装JDK的环境。
1-2）、安装scala
下载：链接：http://pan.baidu.com/s/1nvsB9rF 密码：qy8e 如果无法下载请联系作者。
或者 http://downloads.typesafe.com/scala/2.11.5/scala-2.11.5.tgz
[root@hadoop1 local]# tar -zxvf scala-2.10.6.tgz
[root@hadoop1 local]# vi /etc/profile
加入以下配置：
export SCALA_HOME=/usr/local/scala-2.10.6
[root@hadoop1 local]# source /etc/profile
1-3）、运行
[root@hadoop1 local]# clear
[root@hadoop1 local]# scala
Welcome to Scala version 2.10.6 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80).
Type in expressions to have them evaluated.
Type :help for more information.
scala> print("wordcount")
wordcount
scala> 100+100
res1: Int = 200
scala> print(res1)
200
scala>
scala 基础
1-1）、声明变量
// 整形变量
scala> val int=100
int: Int = 100
// 字符型变量
scala> val string="wordcount"
string: String = wordcount
scala> val string:String="wordcount"
string: String = wordcount
// byte的类型
scala> val byte ='a'
byte: Char = a
// char的类型
scala> val char = 'a'
char: Char = a
// Short的类型
scala> val short = 10L;
short: Long = 10
// Long类型
scala> val long =100L
long: Long = 100
// Float类型
scala> val float =10.1
float: Double = 10.1
// Boolean类型
scala> val boolean = true
boolean: Boolean = true
// double类型
scala> val double = 10.1
double: Double = 10.1
注意在定义数据类型时：val 的缩写是value 数据是不可改变的，var 是可变长度的，在使用时建议使用val 参数，因为在JVM垃圾回收时会更方便。
1-2）、表达式
第一种写法：
def main(args: Array[String]): Unit = {
val index = 20
if (index > 10) {
println("这个数大于10")
} else {
println("这个数小于10")
}
}
第二种写法：
val x = 10
// y直接接受返回的值
val y = {
if (x > 10) {
100
} else if (x > 10 && x < 1000) {
1000
} else {
"error"
}
}
println(y)
第三种写法：
val x = 10
// y直接接受返回的值
val y =
if (x > 10) {
100
} else if (x > 10 && x < 1000) {
1000
} else {
"error"
}
println(y)
def main(args: Array[String]): Unit = {
val value = if (index == 20) {
"ok"
} else {
"flase"
}
print("value:" + value)
}
def main(args: Array[String]): Unit = {
// Array 的迭代
var vls = Array("34", "43", "343", "43");
for (y <- vls) {
println("y:" + y);
}
// 可以把两套的循环放到一个for里面,与JAVA的for一样
for (i <- 1 to 3; y <- 1 to 5) {
println(i + "-" + y)
}
// yield :表示对之前的数据进行操作，RDD复杂度取决于前面的数据。
val ind = for (i <- 1 to 10) yield i * 10;
println(ind);
// 迭代
val arr = Array(1, 2, 3, 4, 5, 6, 7, 8)
for (x <- arr) {
print(x)
}
// 反转
for (i <- (0 until arr.length).reverse) {
print(arr(i))
}
}
// until 是前闭后开的一个关键字
val a = Array("a", "b", "c")
for (i <- 0 to a.length) println(i)
println("===================")
for (i <- 0 until a.length) println(i)
1-3）、Scala运算符
Scala支持与JAVA一样的运算符，实例如下：
scala> val a = 1
a: Int = 1
scala> val b = 2
b: Int = 2
scala> val c = a+(b)
c: Int = 3
scala> print(c)
3
scala> val str = "wordcount"
str: String = wordcount
scala> print(c+str)
3wordcount
1-4）、方法和函数
object ScalaTest {
def function1() {
println("方法一")
}
// unit 相当于没有返回值，相当于JAVA上的void
def function2(): Unit = {
println("定义方法二")
}
def function3(a: Int, b: Int): String = {
"定义方法三:" + (a + b)
}
def function4() = {
println("定义方法四")
}
def function5(): Int = {
10 + 1
}
def function6(a: Int, b: Int) = {
val value = a + b
println(value)
}
def main(args: Array[String]): Unit = {
function1
function2
println(function3(1, 2))
function4
println(function5)
function6(1, 2)
}
}
方法体是可以不写的，回自动的推断出来，但是对于递归的方法来说是必须要写的。
方法一：
object ScalaTest {
def main(args: Array[String]): Unit = {
val index = (x: Int, y: Int) => x + y
println(index(1, 2))
val index1 = (x: Int, y: Int) => {
x + y
}
println(index1(1, 3))
}
}
方法二：
// 定义原始数组
val arr = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
// 第一种匿名函数有数据类型的限制
val res = arr.map((x: Int) => x * 10)
println(res.toBuffer)
// 第二种匿名函数有数据类型的限制并把逻辑放在大括号里面
val re1 = arr.map((x: Int) => {
x * 10
})
println(re1.toBuffer)
// 因为有了数据类型的限制直接使用参数
val re2 = arr.map(x => x * 10)
println(re2.toBuffer)
// 使用神奇的下划线代表arr中的元素
val re3 = arr.map(_ * 10)
println(re3.toBuffer)
// 把函数传到方法里面
val f1 = (x: Int) => x * 10
println(arr.map(f1).toBuffer)
// 过滤需要的数据
val re4 = arr.filter(_ % 2 == 0)
println("filter:" + re4.toBuffer)
方法三：
// 定义最终的函数
val fun: Int => Int = { x => x * 10 }
println(fun(5))
// 多个参数的实例
val f3: (Int, Double) => (Double, Int) = { (x, y) => (y, x) }
val r = f3(3, 3.0)
println(r)
//在一个数据集中可以直接使用点的形式获取数据
println(r._2)
可以看出下划线可以代表集合中的元素，方法以及函数。
可以看出在定义方法时用def来定义而函数则用val或var来定义，方法一般有返回值，而函数则没有，Scala是函数式编程，所以在scala中可以把一切转化为函数，起到了至关的重要性。
函数实例一：
scala> var f1 = (x: Int, y: Int) => x * y;
f1: (Int, Int) => Int = <function2>
scala> print(f1(1,333))
333
函数实例二：
scala> var f2 = (x:Int,y:Int) => { x+ y }
f2: (Int, Int) => Int = <function2>
scala> print(f2(33,5667))
5700
scala> def m1(f2: (Int, Int) => Int): Int = {
| f2(2, 6)
| }
m1: (f2: (Int, Int) => Int)Int
scala> var d1 = m1 _
d1: ((Int, Int) => Int) => Int = <function1>
scala> print(d1)
<function1>
package TestScala
object Test {
// 定义一个方法
def fff(f: Int => Int): Unit = {
val r = f(10)
println(r)
}
// 定义一个函数
val f0 = (x: Int) => {
x * x
}
// 定义一个方法
def m0(x: Int): Int = {
x * 10
}
def main(args: Array[String]): Unit = {
// 把函数传递到方法里面
fff(f0)
// 把方法传入到方法里面
fff(m0)
fff(m0 _)
fff((x: Int) => m0(x))
}
}
1-5）、集合--映射
1-1）、整数
scala> var arr1 = Array(1,2,445);
arr1: Array[Int] = Array(1, 2, 445)
scala> println(arr1.toBuffer)
ArrayBuffer(1, 2, 445)
scala> println(arr1(1));
2
1-2）、字符串
scala> val arr3 = Array("hadoop", "storm", "scala")
arr3: Array[String] = Array(hadoop, storm, scala)
scala> println(arr3.toBuffer)
ArrayBuffer(hadoop, storm, scala)
scala> println(arr3(2))
scala
1-3）、混合
scala> val arr4 = Array("hadoop", "storm", "scala",1,2)
arr4: Array[Any] = Array(hadoop, storm, scala, 1, 2)
scala> print(arr4.toBuffer)
ArrayBuffer(hadoop, storm, scala, 1, 2)
scala> print(arr4(3))
1
scala> import scala.collection.mutable.ArrayBuffer
import scala.collection.mutable.ArrayBuffer
scala> val ab = ArrayBuffer[Int](6);
ab: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(6)
scala> println(ab.toBuffer)
ArrayBuffer(6)
1-1）、循环
yield关键字将原始的数组进行转换会产生一个新的数组，原始的数组不变
val arrq = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
val res = for (e <- arrq if e % 2 == 0) yield e * 10
println(res.toBuffer)
val r = arrq.filter(_ % 2 == 0).map(_ * 10)
println(r.toBuffer)
1-2）、数学运算
val arr = Array(1, 2, 3, 4, 5, 643, 333)
// 求和
println(arr.sum)
// 查找最大值
println(arr.max)
// 查找最小值
println(arr.min)
// 对数据进行反转
println(arr.sorted.reverse)
// 对数据进行正序排序
println(arr.sorted.toBuffer)
// 对数据进行倒叙排序
println(arr.sortWith((x, y) => x > y).toBuffer)
println(arr.sortWith(_ > _).toBuffer)
// 对数据进行累加
println(arr.reduce(_ + _))
println(arr.reduce((x: Int, y: Int) => { x + y }))
1-3）、Map数组的映射
// 这样可以看做成元组的形式
val map = Map("a" -> 1, "b" -> 2, "c" -> 3)
val map1 = Map(("a", 1), ("b", 2), ("c", 3))
// 以下方式在命令行中可用
map += ("d" -> 4)
map += (("e", 5))
// 获取a的值，返回的是Some
val maybeInt: Option[Int] = map.get("a")
println(maybeInt)
// 获取a的值
println(map("a"))
// 获取元素如果没有数据则显示备注的值
println(map.getOrElse("d", "没有找到"))
// 查看集合的大小
println(map.size)
// 查看是否是实体
println(map.empty)
D）、List 操作
val li = List(1, 2, 3, 4, 4, 5, 6, 7)
// 在集合的头部追加元素
val ints: List[Int] = 8 :: li
println(ints.toBuffer)
// 往尾部追加数据
val list1: List[Int] = li.::(8)
print(list1)
// 构建list集合
val list = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
// 对每个元素扩大10倍
list.map((x: Int) => {
x * 10
})
val map: List[Int] = list.map((x: Int) => x * 10)
println("map = " + map)
val map1: List[Int] = list.map(_ * 10)
println("map1 = " + map1)
//转化为list
val toList: List[Int] = list.map((x: Int) => x * 10).toList
println("toList = " + toList)
// 对元素分组
val grouped: Iterator[List[Int]] = list.map((x: Int) => x * 10).grouped(4)
// 转换数据的格式
val toList1: List[List[Int]] = grouped.toList
// 对数据进行压平
val flatten: List[Int] = toList1.flatten
println("flatten = " + flatten)
// 对数据进行先扩大十倍在压平
val map2: List[List[Int]] = toList1.map(_.map(_ * 10))
println("map2 = " + map2)
// 先扩大十倍在压平
val map3: List[Int] = toList1.flatMap(_.map(_ * 10))
println("map3 = " + map3)
// 初始化list集合
val list = ListBuffer(1, 2, 2, 3, 3, 4, 545, 6, 4556)
// 对第一个元素进行重新赋值
list(0) = 100
// 查看结果
println("list = " + list.toBuffer)
// 在list的后面追加一个元素
list += 8888
println(list.toBuffer)
// 以元组的形式增加数据
list += (1, 8888, 99999)
println(list.toBuffer)
// 在元素的尾部追加数据
list.append(1234567890)
println(list.toBuffer)
// 创建可变的set的集合
val set = new HashSet[Int]()
// 为元素赋值
val ints: HashSet[Int] = set + 1
println(ints.toBuffer)
// 增加元素
val ints1: HashSet[Int] = set ++ Set(1, 2, 3, 4, 5, 6)
println(ints1.toBuffer)
val ints2: Set[Int] = Set(7, 8, 9) ++ set
println(ints2.toBuffer)
val add: Boolean = set.add(88888888)
println(add.toString)
// 删除元素
set.remove(2)
println("set = " + set)
// 创建可变长度的hashMao
val map = new mutable.HashMap[String, Int]()
// 增加元素
map.put("xiaozhang", 1)
println("map = " + map)
map("xiaowang") = 2
map += (("xiaoli", 3))
println("map = " + map)
import scala.Function4
import scala.collection.mutable.HashMap
object ScalaTest {
def main(args: Array[String]): Unit = {
val list0 = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 0);
println(list0.toBuffer)
println(list0.toBuffer.sorted)
println(list0.toBuffer.sorted.reverse)
println(list0.toBuffer.sorted.reverse.grouped(4).toList)
println(list0.toBuffer.sorted.reverse.grouped(4).toList.flatten)
println(list0.toBuffer.sorted.reverse.grouped(4).toList.flatMap(_.map(_ * 10)));
println("-------------------------------------")
// heoolword
val lines = Array("hellow hell", "xiaozhang xiaowang", "xiao zhang", "da xiao budabuxiao", "xiao xiao xiao xiao");
println(lines.flatMap(_.split(" ")).map((_, 1)).groupBy(_._1).map(t => (t._1, t._2.size)).toList.sortBy(_._2).toList);
println("-------------------------------------")
// flatMap的使用
var llss = Array("wd vw d ew v", "ee ede eee fv de s", "sd dss sd sd sf", "ww sdcsw www");
var sdfg = llss.flatMap(_.split(" "));
println("sdfg:" + sdfg.toBuffer.grouped(4));
// val sf = sdfg.toBuffer.grouped(4);
println("grouped:" + llss.toBuffer.sorted);
println(llss.toBuffer.reverse)
println("-------------------------------------")
}
}
// 构造数据
val lines = List("hello xiaozhao", "hello xiaoxu", "hello xiaownag", "hello xiaozhang")
// 对数据进行分割并进行压平
val map: List[String] = lines.flatMap(_.split(" "))
// 对数据进行统计
val map1: List[(String, Int)] = map.map((_, 1))
// 对每个元组进行分组
val by: Map[String, List[(String, Int)]] = map1.groupBy(_._1)
// 对数据进行累加
val stringToInt: Map[String, Int] = by.map(t => (t._1, t._2.size))
// 对数据进行正序排序
val by1: List[(String, Int)] = stringToInt.toList.sortBy(_._2)
// 对数据进行倒叙排序
val reverse: List[(String, Int)] = by1.reverse
println("reverse = " + reverse)
1-6）、元组
有提示
var value, (s, y, z) = (2, 3, 5)
println(value._1)
没有提示
val index = (1, 2, 3)
println(index._1)
val t = (1, 3.14, "Fred")
println(t._2)
val t = (1, 3.14, "Fred")
val t = (1, 3.14, 67)
t.productIterator.foreach {
i => println("Value = " + i)
}
val t = new Tuple2("www.google.com", "www.runoob.com")
println("反转后的元组: " + t.swap)
1-7）、 类、对象、继承、超类
Scala的类与Java、C++的类比起来更简洁，学完之后你会感觉Scala更好玩、、
package day2scala
/**
* 在Scala中，类并不用声明为public。
* Scala源文件中可以包含多个类，所有这些类都具有公有可见性。
*/
class Persion {
// 用val修饰的变量是只读属性，有getter但没有setter,（相当与Java中用final修饰的变量）
var id = "xiaozhang"
// 用var修饰的变量既有getter又有setter
var age: Int = 18
// 类私有字段,只能在类的内部使用
private var name: String = "xiaowang"
// 对象私有字段,访问权限更加严格的，Person类的方法只能访问到当前对象的字段
private[this] val age = "48"
}
object Persion {
def main(args: Array[String]) {
var persion = new Persion
println(persion.id)
println(persion.name)
println(persion.age)
}
}
xiaozhang
xiaowang
18
package day2scala
/**
* 在Scala中，类并不用声明为public。
* Scala源文件中可以包含多个类，所有这些类都具有公有可见性。
*/
class Persion {
// 用val修饰的变量是只读属性，有getter但没有setter,（相当与Java中用final修饰的变量）
var id = "xiaozhang"
// 用var修饰的变量既有getter又有setter
var age: Int = 18
// 类私有字段,只能在类的内部使用
private var name: String = "xiaowang"
// 对象私有字段,访问权限更加严格的，Person类的方法只能访问到当前对象的字段
private[this] val age = "48"
}
/**
* 与属性的类的名字一样，并在同一个包中
*/
object Persion {
def main(args: Array[String]) {
var persion = new Persion
println(persion.id)
println(persion.name)
println(persion.age)
}
}
注意：主构造器会执行类定义中的所有语句
package day2scala
class Student(var name: String, var age: Int, faceValue: Double = 99.9, private var height: Int = 20) {
private[this] var gender: String = null
def show() {
println(faceValue)
}
//辅助构造器 def this （参数）
def this(name: String, age: Int, gender: String) {
//辅助构造器第一行一定要先调用主构造器
this(name, age)
this.gender = gender
}
}
object Student {
def main(args: Array[String]) {
var student = new Student("小王", 18, 85, 180)
println(student.age + "-" + student.height + "-" + student.name + "-" + student.show)
student.age = 14;
student.height = 852;
student.name = "xiaoli"
student.show
println(student.age + "-" + student.height + "-" + student.name + "-" + student.show)
}
}
object Main {
def main(args: Array[String]) {
var student = new Student("小王", 18, 85, 180)
println(student.age + "-" + student.name + "-" + student.show)
}
}
注意：访问的范围的问题
Private / private[this] 、、、、
1-1）、 单例对象
package day2scala
class SingletonDemo {
SingletonDemo.sayHi()
sayHi()
var name: String = "xiaozhang"
def sayHi() {
println("这是第一个类的方法")
}
}
object SingletonDemo {
var name: String = "xiaozhang"
def sayHi() {
println("这是第二类的方法")
}
def main(args: Array[String]) {
// 一下调用的是本类的属性
var sing = SingletonDemo
sing.sayHi()
sing.name
// 调用的是SingletonDemo的属性
var singSin = new SingletonDemo
singSin.sayHi()
singSin.name
}
}
1-2）、 应用程序对象
package day3scala
/**
* 运行main方法的另外一种方式
*
* 继承 App --> main 方法
*/
object Dau extends App {
println("sf")
}
/** The main method.
* This stores all argument so that they can be retrieved with `args`
* and the executes all initialization code segments in the order they were
* passed to `delayedInit`
* @param args the arguments passed to the main method
*/
def main(args: Array[String]) = {
this._args = args
for (proc <- initCode) proc()
if (util.Properties.propIsSet("scala.time")) {
val total = currentTime - executionStart
Console.println("[total " + total + "ms]")
}
}
package day3scala
/**
* apply方法的使用
*/
object Student {
def apply() = {
println("这是无惨的apply方法")
}
def apply(name: String): String = {
println("这是带一个参数的apply的方法" + name + "")
// 返回的类型
name
}
def apply(name: String, sex: Int) = {
println("这是既有String又有Int类型的apply的方法")
}
def main(args: Array[String]) {
Student("小张")
}
}
在scala中扩展类的方式与咱们老朋友JAVA的实现方式是一样一样的，在scala
中定义接口的关键字是Trait
package day3scala
/**
* 一个类可以多实现但是只能单继承，就像一个小孩的特征像爸爸又像他二叔以及三叔
*/
class classDome {
def say(name: String) = {
println("classDome 的：" + name)
}
}
trait ClassDomeTrait {
def classDomeRraitTest(name: String): String = {
"ClassDomeTrait已被调用" + name
}
}
trait ClassDomeTrait1 {
def ClassDomeTrait1Test = {
println("ClassDomeTrait1Test已被调用")
}
}
object ClassDomeDay3 extends classDome with ClassDomeTrait with ClassDomeTrait1 {
def main(args: Array[String]) {
say("小张")
var test = classDomeRraitTest("大哥大")
println(test)
ClassDomeTrait1Test
}
}
1-8）、 模式匹配
package day2scala.cases
import scala.util.Random
case class Dome3(name: String, higth: String)
case class Dome4(x: Int, y: Int)
case class Dome5(z: Int)
/**
* 匹配字符串可以借助第三方的类来实现参数的传递与获取，这是JAVA比不了的
*/
object Dome2 {
def main(args: Array[String]) {
val arr = Array(Dome3("333", "43"), Dome4(3, 5), Dome5(5));
// 主要取值的范围
arr(Random.nextInt(arr.length)) match {
case Dome3(name, higth) => {
println("dome3")
}
case Dome4(3, 4) => {
println("Dome4")
}
case Dome5(5) => {
println("exit")
}
}
val map = Map("a" -> 1, "b" -> 2, "c" -> 3)
val v = map.get("a") match {
case Some(i) => println("-------------------")
case None => println("++++++++++++++++++++++++++")
}
def func2(num: String): Int = num match {
case "one" => 1
case "two" => 2
case _ => -1
}
println(func2("one"))
// 偏函数
def func1: PartialFunction[String, Int] = {
case "one" => {
println("one case")
1
}
case "two" => 2
case _ => -1
}
println(func1("one"))
}
}
package day2scala.cases
import scala.util.Random
case class Dome(name: String)
/**
* 字符串匹配加类型匹配
*/
object Dome1 {
def main(args: Array[String]) {
val arr = Array("校长", "小王", "小李", "大和大", "回复", 3, 343.6, 'd', true)
val name = arr(Random.nextInt(arr.length))
println(name)
name match {
case "校长" => println("")
case "xiaowang" => {
println("xiaozhang")
}
case "小李" => {
println("小李")
}
case x: Int if (x > 10) => {
println("Int:" + x)
}
case y: Double => {
println("double:" + y)
}
case xx: Int if (xx > 20) => {
throw new Exception("你输入的数大于了20")
}
case ("小_", "", "") => {
println("小_")
}
case (_, _, "小_") => {
println("________小")
}
// _ 如果没有匹配则会执行这段代码
case _ => {
println("exit")
exit
}
}
}
}
package day2scala.cases
/**
* 集合的匹配
*/
object DomeTest {
def main(args: Array[String]) {
val arr = Array(1, 1, 7, 0)
arr match {
case Array(1, 1, x, y) => println(x + ":" + y)
// case Array(1, 1, 7, y) => println("only 0")
case Array(0, _*) => println("0 ...")
case _ => println("something else")
}
val lst = List(0)
lst match {
case 0 :: Nil => println("only 0")
case x :: y :: Nil => println("x $x y $y")
case 0 :: a => println("0 ... $a")
case _ => println("something else")
}
}
}
package day3scala
/**
* 在scala中option中表示类型可能存在或者可能不存在的值，如果存在则用some表示，没有则用None表示
*/
object OptionDome {
def main(args: Array[String]) {
var arr = Map("xiaozhang" -> 2, "xiaowang" -> 4)
var ls = arr.get("xiaozhang") match {
case Some(i) => {
println(i)
}
case None => {
println("没有找到")
}
}
// 更好的实现方式
var ld = arr.getOrElse("c", 0)
println(ld)
}
}
被花括号括起来的但是没有match的一组叫做偏函数
def func1: PartialFunction[String, Int] = {
case "one" => {
println("one case")
1
}
case "two" => 2
case _ => -1
}
println(func1("one"))
Scala 在win下编译scala文件
1-1）、准备数据
object TestCount {
def main(args: Array[String]): Unit = {
System.out.print("hellword Scala")
}
}
1-2）、进行编译并查看结果
IDEA开发Scala程序
1-1）、准备IDEA并安装scala插件
1-2）、创建scala项目
1-3）、把项目导出来运行
1-4）、运行查看效果
Scala 的Actor 编程(也就是Akka的编程)
! 发送异步消息，没有返回值。
!? 发送同步消息，等待返回值。
!! 发送异步消息，有返回值
react方式会复用线程，比receive更高效
实例一：
package day2scala.actor
import scala.actors.Actor
/**
* MyActorTest执行的时候首先调用start方法
*/
object MyActorTest extends Actor {
def act() {
for (i <- 1 to 10) {
println("MyActorTest:" + Thread.currentThread().getName())
Thread.sleep(1000)
}
}
}
object MyActorTest1 extends Actor {
def act() {
for (i <- 1 to 10) {
println("MyActorTest1:" + Thread.currentThread().getName())
Thread.sleep(1000)
}
}
}
object MyActorTest2 {
def main(args: Array[String]) {
MyActorTest.start
MyActorTest1.start
}
}
实例二：
package day2scala.actor
import scala.actors.Actor
/**
* loop 是个无限循环的，相当于oracle上的loop
*/
class YourActorText extends Actor {
def act() {
loop {
react {
case "start" => {
println("staring .......")
println(Thread.currentThread().getName())
}
case "end" => {
println("end ......")
println(Thread.currentThread().getName())
}
case "exit" => {
exit()
}
}
}
}
}
object YourActorText1 {
def main(args: Array[String]) {
var your = new YourActorText
your.start
your.!("start")
your.!!("exit")
}
}
实例三：
package day2scala.actor
import scala.actors.Actor
import scala.io.Source
import scala.actors.Future
import scala.collection.mutable.HashSet
import scala.collection.mutable.ListBuffer
case class SubmitTask(filePath: String)
case class ResultTask(result: Map[String, Int])
case object StopTask
/**
* loop 是无限循环
* react 比receive快
*/
class Task extends Actor {
def act() {
loop {
react {
case SubmitTask(filePath) => {
val result = Source.fromFile(filePath).getLines.toList.flatMap(_.split(" ")).map((_, 1)).groupBy(_._1).mapValues(_.size)
sender ! ResultTask(result)
}
case StopTask => {
exit()
}
}
}
}
}
object ActorWordCount {
def main(args: Array[String]) {
val replySet = new HashSet[Future[Any]]()
val resultList = new ListBuffer[ResultTask]()
val list = Array[String]("D:\\text.txt", "D:\\text1.txt", "D:\\text2.txt")
for (filePath <- list) {
val task = new Task
val reply = task.start !! SubmitTask(filePath)
replySet += reply
}
while (replySet.size > 0) {
// _.isSet 判断是不是Set集合
val toCompute = replySet.filter(_.isSet)
for (f <- toCompute) {
// 把执行的结果付给ResultTask
val result = f().asInstanceOf[ResultTask]
resultList += result
replySet -= f
}
Thread.sleep(1000)
}
val fr = resultList.flatMap(_.result).groupBy(_._1).mapValues(_.foldLeft(0)(_ + _._2))
println(fr)
exit
}
}
Scala 高级特性
1-1 )、函数的总结
package day3scala
/**
* 柯里化的函数感觉挺有意思的
*/
object Function {
def main(args: Array[String]) {
val arr = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 0)
// 定义一个函数并把函数复制给了fun1
val fun1 = (x: Int) => x * 10
// 把函数作为参数传入到MAP中
arr.map(fun1)
// 匿名函数，直接把结果传入到map中，效果与以前的一样
arr.map((x: Int) => x * 10)
// 使用scala的推断类型的方式来进行计算
arr.map(x => x * 10)
// 神奇的下划线
arr.map(_ * 10)
// 定义一个方法
def m(x: Int) = x * 10
// 把方法转化为了函数
val mm = m _
// 讲方法传入到map中
arr.map(mm)
for (x <- arr) {
println(x)
}
// 柯里化的转换
def ss(x: Int) = (yy: Int) => x * yy
// 转化为函数,传递第一个参数
val ff = ss(4)
// 传递第二个参数
val asd = ff(9)
// 或者这样写
val dfg = ss(3)(8)
println(dfg)
}
}
1-2）、隐式转换
scala> :implicit -v
/* 78 implicit members imported from scala.Predef */
/* 48 inherited from scala.Predef */
implicit def any2ArrowAssoc[A](x: A): ArrowAssoc[A]
implicit def any2Ensuring[A](x: A): Ensuring[A]
implicit def any2stringadd(x: Any): runtime.StringAdd
implicit def any2stringfmt(x: Any): runtime.StringFormat
implicit def boolean2BooleanConflict(x: Boolean): Object
implicit def byte2ByteConflict(x: Byte): Object
implicit def char2CharacterConflict(x: Char): Object
implicit def double2DoubleConflict(x: Double): Object
implicit def float2FloatConflict(x: Float): Object
implicit def int2IntegerConflict(x: Int): Object
implicit def long2LongConflict(x: Long): Object
implicit def short2ShortConflict(x: Short): Object
implicit def booleanArrayOps(xs: Array[Boolean]): mutable.ArrayOps[Boolean]
implicit def byteArrayOps(xs: Array[Byte]): mutable.ArrayOps[Byte]
implicit def charArrayOps(xs: Array[Char]): mutable.ArrayOps[Char]
implicit def doubleArrayOps(xs: Array[Double]): mutable.ArrayOps[Double]
implicit def floatArrayOps(xs: Array[Float]): mutable.ArrayOps[Float]
implicit def genericArrayOps[T](xs: Array[T]): mutable.ArrayOps[T]
implicit def intArrayOps(xs: Array[Int]): mutable.ArrayOps[Int]
implicit def longArrayOps(xs: Array[Long]): mutable.ArrayOps[Long]
implicit def refArrayOps[T <: AnyRef](xs: Array[T]): mutable.ArrayOps[T]
implicit def shortArrayOps(xs: Array[Short]): mutable.ArrayOps[Short]
implicit def unitArrayOps(xs: Array[Unit]): mutable.ArrayOps[Unit]
implicit def Boolean2boolean(x: Boolean): Boolean
implicit def Byte2byte(x: Byte): Byte
implicit def Character2char(x: Character): Char
implicit def Double2double(x: Double): Double
implicit def Float2float(x: Float): Float
implicit def Integer2int(x: Integer): Int
implicit def Long2long(x: Long): Long
implicit def Short2short(x: Short): Short
implicit val StringCanBuildFrom: generic.CanBuildFrom[String,Char,String]
implicit def arrayToCharSequence(xs: Array[Char]): CharSequence
implicit def augmentString(x: String): immutable.StringOps
implicit def boolean2Boolean(x: Boolean): Boolean
implicit def byte2Byte(x: Byte): Byte
implicit def char2Character(x: Char): Character
implicit def conforms[A]: <:<[A,A]
implicit def double2Double(x: Double): Double
implicit def exceptionWrapper(exc: Throwable): runtime.RichException
implicit def float2Float(x: Float): Float
implicit def int2Integer(x: Int): Integer
implicit def long2Long(x: Long): Long
implicit def seqToCharSequence(xs: IndexedSeq[Char]): CharSequence
implicit def short2Short(x: Short): Short
implicit def tuple2ToZippedOps[T1, T2](x: (T1, T2)): runtime.Tuple2Zipped.Ops[T1,T2]
implicit def tuple3ToZippedOps[T1, T2, T3](x: (T1, T2, T3)): runtime.Tuple3Zipped.Ops[T1,T2,T3]
implicit def unaugmentString(x: immutable.StringOps): String
/* 30 inherited from scala.LowPriorityImplicits */
implicit def genericWrapArray[T](xs: Array[T]): mutable.WrappedArray[T]
implicit def wrapBooleanArray(xs: Array[Boolean]): mutable.WrappedArray[Boolean]
implicit def wrapByteArray(xs: Array[Byte]): mutable.WrappedArray[Byte]
implicit def wrapCharArray(xs: Array[Char]): mutable.WrappedArray[Char]
implicit def wrapDoubleArray(xs: Array[Double]): mutable.WrappedArray[Double]
implicit def wrapFloatArray(xs: Array[Float]): mutable.WrappedArray[Float]
implicit def wrapIntArray(xs: Array[Int]): mutable.WrappedArray[Int]
implicit def wrapLongArray(xs: Array[Long]): mutable.WrappedArray[Long]
implicit def wrapRefArray[T <: AnyRef](xs: Array[T]): mutable.WrappedArray[T]
implicit def wrapShortArray(xs: Array[Short]): mutable.WrappedArray[Short]
implicit def wrapUnitArray(xs: Array[Unit]): mutable.WrappedArray[Unit]
implicit def Boolean2booleanNullConflict(x: Null): Boolean
implicit def Byte2byteNullConflict(x: Null): Byte
implicit def Character2charNullConflict(x: Null): Char
implicit def Double2doubleNullConflict(x: Null): Double
implicit def Float2floatNullConflict(x: Null): Float
implicit def Integer2intNullConflict(x: Null): Int
implicit def Long2longNullConflict(x: Null): Long
implicit def Short2shortNullConflict(x: Null): Short
implicit def booleanWrapper(x: Boolean): runtime.RichBoolean
implicit def byteWrapper(x: Byte): runtime.RichByte
implicit def charWrapper(c: Char): runtime.RichChar
implicit def doubleWrapper(x: Double): runtime.RichDouble
implicit def fallbackStringCanBuildFrom[T]: generic.CanBuildFrom[String,T,immutable.IndexedSeq[T]]
implicit def floatWrapper(x: Float): runtime.RichFloat
implicit def intWrapper(x: Int): runtime.RichInt
implicit def longWrapper(x: Long): runtime.RichLong
implicit def shortWrapper(x: Short): runtime.RichShort
implicit def unwrapString(ws: immutable.WrappedString): String
implicit def wrapString(s: String): immutable.WrappedString
object Predef extends LowPriorityImplicits
多看一下Predef类下面的方法，可以看出导入了基本的数据类型以及集合类型
C）、查看LowPriorityImplicits类下的隐式方法
class LowPriorityImplicits {
@inline implicit def byteWrapper(x: Byte) = new runtime.RichByte(x)
@inline implicit def shortWrapper(x: Short) = new runtime.RichShort(x)
@inline implicit def intWrapper(x: Int) = new runtime.RichInt(x)
@inline implicit def charWrapper(c: Char) = new runtime.RichChar(c)
@inline implicit def longWrapper(x: Long) = new runtime.RichLong(x)
@inline implicit def floatWrapper(x: Float) = new runtime.RichFloat(x)
@inline implicit def doubleWrapper(x: Double) = new runtime.RichDouble(x)
@inline implicit def booleanWrapper(x: Boolean) = new runtime.RichBoolean(x)
}
可以看出导入了许多隐式的方法，其中intWrapper方法就是实现了to方法的实现
package Rich
import java.io.File
/**
* Created by Administrator on 2016/12/3.
*/
object MyPreDef {
// 定义隐式转换方法（将原来的类型转换为增强的类型）
implicit def file2RichFile(file: File) = new RichFile(file)
}
package Rich
import java.io.File
import scala.io.Source
class RichFile(file: File) {
def read: String = Source.fromFile(file).mkString
}
object RichFile {
def main(args: Array[String]) {
val file = new File("D:\\china.txt")
// 装饰模式显示的增强
val rf = new RichFile(file)
val read: String = rf.read
println("read = " + read)
// 隐式增强的方式
val file1 = new File("D:\\china.txt")
import MyPreDef.file2RichFile
val read1: String = file1.read
println("read1 = " + read1)
}
}
1-3）、柯里化
// 初始化隐士参数
object MyPreDef {
implicit val aaaa: Int = 1024
}
object CurryingTest {
// 定义一个普通方法
def curry(x: Int, y: Int) = {
x * y
}
// 定义柯里化方法一
def currying(x: Int)(y: Int) = {
x * y
}
// 初始化隐实值
def curryingImp(x: Int)(implicit y: Int = 100) = {
x * y
}
// 定义柯里化方法二
def curryingTest(x: Int) = (y: Int) => {
x * y
}
def main(args: Array[String]) {
// 调用普通方法
val curry1: Int = curry(2, 3)
println("curry1 = " + curry1)
// 调用柯里化的方法
val currying1: Int = currying(2)(6)
println("currying1 = " + currying1)
// 把柯里化的方法转换为函数
val intToInt: (Int) => Int = currying(2) _
// 调用转换的方法
val int: Int = intToInt(8)
println("int = " + int)
// 带有隐士参数的柯里化方法
val curryingImp1: Int = curryingImp(2)
println("curryingImp1 = " + curryingImp1)
// 去MyPreDef查找类型匹配的参数，使用最新的参数，如果找不到则使用默认的
import MyPreDef.aaaa
val imp: Int = curryingImp(2)
println("imp = " + imp)
// 测试柯里化方法二
val test: (Int) => Int = curryingTest(2)
val test1: Int = test(5)
println("test1 = " + test1)
}
}
1-4）、泛型
[ T < : UpperBound ] 上界
[ T >: LowerBound] 下界
[ T <% ViewBound] 视图界定
[ T : ContextBound] 上下界
[ T + ] 斜变
[ -T ] 逆变
实体类
package gen
class Boy(val name: String, val faceValue: Int) extends Comparable[Boy] {
// 添加匹配规则
override def compareTo(o: Boy): Int = {
this.faceValue - o.faceValue
}
}
具体实现的类
package gen
// 在类中定义泛型
class MrRight[T <: Comparable[T]] {
// 第二种方式定义泛型
// def choose[T <: Comparable[T]](first: T, second: T): T = {
// if (first.compareTo(second) > 0) first else second
// }
def choose(first: T, second: T): T = {
if (first.compareTo(second) > 0) first else second
}
}
object MrRight {
def main(args: Array[String]) {
val mr = new MrRight[Boy]
val boy = new Boy("xiaozhang", 88)
val boy1 = new Boy("xiaowang", 8888)
val choose: Boy = mr.choose(boy, boy1)
println("choose = " + choose.name)
}
}
实体类
package gen
// 定义实体类
class Girl(val name: String, val faceValue: Int) {
}
定义隐式方法
package gen
// 定义匹配的规则
object MyPreDef {
// 隐式的方法
implicit def girl2Ordered(g: Girl) = new Ordered[Girl] {
override def compare(that: Girl): Int = {
g.faceValue - that.faceValue
}
}
}
具体实现的类
package gen
// 实现Ordered方法，Ordered重写了比较条件的方法
class MissRight[T <% Ordered[T]] {
def choose(first: T, second: T): T = {
if (first > second) first else second
}
}
object MissRight {
def main(args: Array[String]) {
// 导入规则
import MyPreDef.girl2Ordered
val mr = new MissRight[Girl]
// 实例化对象信息
val g1 = new Girl("xiaolili", 88)
val g2 = new Girl("xiaomoli", 99)
// 调用方法调用
val choose: Girl = mr.choose(g1, g2)
println("choose = " + choose.name)
}
}
视图转换可以把比较规则与实体分离开来，这样就大大的解决了不同的功能的问题。
实体类
package gen
// 定义实体类
class Girl(val name: String, val faceValue: Int) {
}
定义隐式方法
package gen
// 定义匹配的规则
object MyPreDef {
// 上下文转换的值
implicit object girl2Ordering extends Ordering[Girl] {
override def compare(x: Girl, y: Girl): Int = {
x.faceValue - y.faceValue
}
}
}
具体实现的类
package gen
// 上下文的界定不需要Ordering不需要指定泛型
class MissRight[T: Ordering] {
def select(first: T, second: T): T = {
val ord = implicitly[Ordering[T]]
if (ord.gt(first, second)) first else second
}
}
object MissRight {
def main(args: Array[String]) {
// 导入规则
import MyPreDef.girl2Ordering
val mr = new MissRight[Girl]
// 实例化对象信息
val g1 = new Girl("xiaolili", 88)
val g2 = new Girl("xiaomoli", 99)
// 调用方法调用
val choose: Girl = mr.select(g1, g2)
println("choose = " + choose.name)
}
}
Spark-shell 执行Helloword
1-1）、查看启动过程日志信息
[root@hadoop1 bin]# /usr/local/spark-1.3.1-bin-hadoop2.6/bin
[root@hadoop1 bin]# ./spark-shell
16/09/17 23:54:07 INFO DiskBlockManager: Created local directory at /tmp/spark-3f925acf-98d5-43de-b04a-bea37b20bf2d/blockmgr-99460b06-2031-4ac6-b7fd-be2115b3402b
16/09/17 23:54:07 INFO HttpFileServer: HTTP File server directory is /tmp/spark-124e87a9-8e9b-4b90-9627-95ff8dfe67d0/httpd-6390a167-dd6a-49b7-8e95-1b0718dd860c
16/09/17 23:54:08 INFO SparkUI: Started SparkUI at http://hadoop1:4040
1-2）、查看成成的文件信息：
[root@hadoop1 /]# cd tmp/
spark-3f925acf-98d5-43de-b04a-bea37b20bf2d/blockmgr-99460b06-2031-4ac6-b7fd-be2115b3402b
spark-124e87a9-8e9b-4b90-9627-95ff8dfe67d0/httpd-6390a167-dd6a-49b7-8e95-1b0718dd860c
1-3）、计算wordCount
scala> sc.textFile("/usr/local/words.txt")
res0: org.apache.spark.rdd.RDD[String] = /usr/local/words.txt MapPartitionsRDD[1] at textFile at <console>:22
scala> sc.textFile("/usr/local/words.txt").flatMap(_.split(" "))
res2: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[4] at flatMap at <console>:22
scala> sc.textFile("/usr/local/words.txt").flatMap(_.split(" ")).map((_,1))
res3: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[8] at map at <console>:22
scala> sc.textFile("/usr/local/words.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
res5: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[17] at reduceByKey at <console>:22
scala> sc.textFile("/usr/local/words.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect
res6: Array[(String, Int)] = Array((dageda,1), (xiaoli,1), (hellow,4), (xisdsd,1), (xiaozhang,1))
1-4）、查看spark默认的指定的分区数
scala> sc.parallelize(1 to 10, 2)
res1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:22
scala> res1.partitions.size
res3: Int = 2
scala> sc.parallelize(1 to 10)
res4: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at <console>:22
scala> res4.partitions.size
res5: Int = 1
系统根据CPU的核数来分配分区的数量
Wordcount 实例
package day2scala.actor
import scala.actors.Actor
import scala.io.Source
import scala.actors.Future
import scala.collection.mutable.HashSet
import scala.collection.mutable.ListBuffer
case class SubmitTask(filePath: String)
case class ResultTask(result: Map[String, Int])
case object StopTask
/**
* loop 是无限循环
* react 比receive快
*/
class Task extends Actor {
def act() {
loop {
react {
case SubmitTask(filePath) => {
val result = Source.fromFile(filePath).getLines.toList.flatMap(_.split(" ")).map((_, 1)).groupBy(_._1).mapValues(_.size)
sender ! ResultTask(result)
}
case StopTask => {
exit()
}
}
}
}
}
object ActorWordCount {
def main(args: Array[String]) {
val replySet = new HashSet[Future[Any]]()
val resultList = new ListBuffer[ResultTask]()
val list = Array[String]("D:\\text.txt", "D:\\text1.txt", "D:\\text2.txt")
for (filePath <- list) {
val task = new Task
val reply = task.start !! SubmitTask(filePath)
replySet += reply
}
while (replySet.size > 0) {
// _.isSet 判断是不是Set集合
val toCompute = replySet.filter(_.isSet)
for (f <- toCompute) {
// 把执行的结果付给ResultTask
val result = f().asInstanceOf[ResultTask]
resultList += result
replySet -= f
}
Thread.sleep(1000)
}
val fr = resultList.flatMap(_.result).groupBy(_._1).mapValues(_.foldLeft(0)(_ + _._2))
println(fr)
exit
}
}
Scala 简单RPC实现
1-1）、图解
1-2）、初步实现
import akka.actor.{Actor, ActorRef, ActorSystem, Props}
import com.typesafe.config.ConfigFactory
class Master extends Actor {
override def receive: Receive = {
// Master 发送的消息
case "hello" => {
println("hello")
}
case "start" => {
println("start ....")
}
case "stop" => {
println("stop ....")
}
// 接受客户端的信息
case "connect" => {
println("a client connect .....")
// 发送给Woker发送消息
sender ! "success"
}
}
}
object Master {
def main(args: Array[String]) {
// Master 的host
// val host = args(0)
val host = "127.0.0.1"
// Master的port
// val port = args(1)
val port = 8888
// 拼接字符串
val confStr =
s"""
|akka.actor.provider = "akka.remote.RemoteActorRefProvider"
| |akka.remote.netty.tcp.hostname = "$host"
| |akka.remote.netty.tcp.port = "$port"
""".stripMargin
val conf = ConfigFactory.parseString(confStr)
// 创建ActorSystem，是单利的，监听Actor的运行情况
val actorSystem = ActorSystem("MasterActorSystem", conf)
// 通过actorSystem来创建actor,通过以下两种方式创建
val master: ActorRef = actorSystem.actorOf(Props[Master], "Master")
// val master: ActorRef = actorSystem.actorOf(Props(Master.getClass), "Master")
// 发送消息
// ! 发送消息异步消息，没有返回值。
// !? 发送同步消息，等待返回值。
// !!1 发送异步消息，有返回值
master ! "hello"
master ! "start"
master ! "stop"
// 等待线程推出的信息
actorSystem.awaitTermination()
}
}
import akka.actor.{ Actor, ActorRef, ActorSelection, ActorSystem, Props }
import com.typesafe.config.ConfigFactory
/**
* Created by Administrator on 2016/12/1.
*/
class Worker extends Actor {
// preStart 是在构造器之后，receive之前执行
override def preStart(): Unit = {
// 首先和Master 建立链接，拿到Master的引用，并发送下消息
val selection: ActorSelection = context.actorSelection("akka.tcp://[email protected]:2552/user/Master")
selection ! "connect"
}
// 伴生对象的Receive方法
override def receive: Receive = {
// Woker 发送消息的匹配
case "hello" => {
println("hello")
}
case "start" => {
println("start ....")
}
case "stop" => {
println("stop ....")
}
// 接受Master发送的消息
case "success" => {
println("a msg for master : success")
}
}
}
object Worker {
def main(args: Array[String]) {
// woker的IP
val host = "127.0.0.1"
// woker的端口
val port = 8888
// 拼装字符串
val confStr =
s"""
|akka.actor.provider = "akka.remote.RemoteActorRefProvider"
|akka.remote.netty.tcp.hostname = "$host"
|akka.remote.netty.tcp.port = "$port"
""".stripMargin
// 创建配置工厂
val conf = ConfigFactory.parseString(confStr)
// 创建actor的系统配置
val actorSystem = ActorSystem("WokerActor", conf)
// 设置actor
val actorWoker: ActorRef = actorSystem.actorOf(Props[Worker], "Woker")
actorWoker ! "hello"
actorWoker ! "start"
actorWoker ! "stop"
// 优雅的退出程序
actorSystem.awaitTermination()
}
}
1 - 3）、深入探究
package test.cn.****.test
import akka.actor.Actor
import com.typesafe.config.impl.ConfigString
import com.typesafe.config.ConfigFactory
import akka.actor.ActorSystem
import akka.actor.Props
import scala.collection.mutable
import java.util.UUID
import scala.concurrent.duration._
import scala.collection.mutable.HashMap
import scala.collection.mutable.HashSet
class TestMaster(val host: String, val port: Int) extends Actor {
// 保存wokerId 到TestwokerInfo的映射
val idtoWoker = new HashMap[String, TestWorkerInfo]()
// 保存所有TestWokerInfo的信息
val wokers = new HashSet[TestWorkerInfo]
// 单位秒
val CHECK_INTERVAL = 15000
// 在Receive运行之前运行
override def preStart(): Unit = {
import context.dispatcher
context.system.scheduler.schedule(0 millis, CHECK_INTERVAL millis, self, TestCheckTimeOutWorker)
}
def receive: Receive = {
// 接受woker发送过来的注册信息
case TestRegisterWorker(wokerId, cores, memory) => {
if (!idtoWoker.contains(wokerId)) {
try {
val wokerInfo = new TestWorkerInfo(wokerId, cores, memory)
idtoWoker(wokerId) = wokerInfo
wokers += wokerInfo
sender ! TestRegisterdWorker(s"akka.tcp://${TestMaster.MASTER_SYSTEM}@$host:$port/user/${TestMaster.MASTER_NAME}")
} catch {
case t: Throwable => t.printStackTrace()
}
}
}
// worker 发送给master的心跳信息
case TestHeartbeat(workerId) => {
if (idtoWoker.contains(workerId)) {
val workerInfo = idtoWoker(workerId)
val currentTime = System.currentTimeMillis()
workerInfo.lastHeartbeatTime = currentTime
}
}
case TestCheckTimeOutWorker => {
val currentTime = System.currentTimeMillis()
val deadWorkers = wokers.filter { x => currentTime - x.lastHeartbeatTime > CHECK_INTERVAL }
deadWorkers.foreach {
x =>
idtoWoker -= x.wokerId
wokers -= x
}
println("现在存活的个数为:" + wokers.size)
}
}
}
object TestMaster {
// 定义常量
val MASTER_NAME = "TestMaster"
val MASTER_SYSTEM = "TestMasterActorSystem"
def main(args: Array[String]): Unit = {
val host = "127.0.0.1"
val port = 8888
val confStr = s"""
|akka.actor.provider = "akka.remote.RemoteActorRefProvider"
|akka.remote.netty.tcp.hostname = "$host"
|akka.remote.netty.tcp.port = "$port"
""".stripMargin
val conf = ConfigFactory.parseString(confStr)
val actorSystem = ActorSystem(MASTER_SYSTEM, conf)
val actor = actorSystem.actorOf(Props(new TestMaster(host, port)), MASTER_NAME)
actorSystem.awaitTermination()
}
}
package test.cn.****.test
import akka.actor.Actor
import com.typesafe.config.ConfigFactory
import akka.actor.ActorSystem
import akka.actor.ActorSelection
import akka.actor.Props
import java.util.UUID
import scala.concurrent.duration._
class TestWoker(val cores: Int, val memory: Int, val masterHost: String, val masterPort: Int) extends Actor {
// master 的URL
var masterUrl: String = _
// master的引用
var master: ActorSelection = _
// 定时时间
val HEARTBEAT_INTERVAL = 10000
// woker的ID
val wokerId = UUID.randomUUID().toString()
override def preStart(): Unit = {
master = context.actorSelection(s"akka.tcp://${TestMaster.MASTER_SYSTEM}@$masterHost:$masterPort/user/${TestMaster.MASTER_NAME}")
println("preStart:" + UUID.randomUUID().toString())
// 向master发送注册信息
master ! TestRegisterWorker(wokerId, cores, memory)
}
def receive: Receive = {
// 注册完成后master发送给woker的信息
case TestRegisterdWorker(masterUrl) => {
this.masterUrl = masterUrl
// 启动定时任务，定时的向master发送心跳
import context.dispatcher
context.system.scheduler.schedule(0 milli, HEARTBEAT_INTERVAL milli, self, TestSendHeartbeat)
}
// 定时向master发送心跳
case TestSendHeartbeat => {
println("TestSendHeartbeat:" + UUID.randomUUID().toString())
master ! TestHeartbeat(wokerId)
}
}
}
object TestWoker {
def main(args: Array[String]): Unit = {
// woker的信息
val host = "127.0.0.1"
val port = 9999
val cores = 4
val memory = 1024
// master的信息
val masterHost = "127.0.0.1"
val masterPort = 8888
val confStr = s"""
|akka.actor.provider = "akka.remote.RemoteActorRefProvider"
|akka.remote.netty.tcp.hostname = "$host"
|akka.remote.netty.tcp.port = "$port"
""".stripMargin
val conf = ConfigFactory.parseString(confStr)
val actorSystem = ActorSystem("TestWokerActorSystem", conf)
val actor = actorSystem.actorOf(Props(new TestWoker(cores, memory, masterHost, masterPort)), "TestWoker")
actorSystem.awaitTermination()
}
}
package test.cn.****.test
// 定义接口
trait TestMessage extends Serializable
// woker -> master 发送注册信息
case class TestRegisterWorker(workerId: String, cores: Int, memory: Int) extends TestMessage
// master -> woker 注册成功信息
case class TestRegisterdWorker(masterUrl: String) extends TestMessage
// woker 向 master 发送心跳
case class TestHeartbeat(wokerId: String)
// woker 初始化的信息
case object TestSendHeartbeat extends TestMessage
// 检查超时的时间
case object TestCheckTimeOutWorker extends TestMessage
package test.cn.****.test
class TestWorkerInfo(val wokerId: String, val cores: Int, val memory: Int) {
var lastHeartbeatTime: Long = _
}
PageRank 算法计算权重
package com.sdcet
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
/**
* Created by Administrator on 2017/2/10.
*/
object PageRank {
def main(args: Array[String]) {
System.setProperty("hadoop.home.dir", "E:\\winutils-hadoop-2.6.4\\hadoop-2.6.4")
val conf = new SparkConf().setAppName("PageRank").setMaster("local")
val sc = new SparkContext(conf)
val links: RDD[(String, Array[String])] = sc.parallelize(Array(("A", Array("d")), ("B", Array("A")), ("C", Array("A", "B")), ("D", Array("A", "C"))), 2)
var rank: RDD[(String, Double)] = sc.parallelize(Array(("A", 1.0), ("B", 1.0), ("C", 1.0), ("D", 1.0)), 2)
for (i <- 1 to 10) {
val contribs = links.join(rank, 2).flatMap {
case (url, (links, rank)) => links.map(dest => (dest, rank / links.size))
}
rank = contribs.reduceByKey(_ + _, 2).mapValues(0.15 + 0.85 * _)
}
println("rank:"+rank)
rank.saveAsTextFile("hdfs://hadoop1:9000/pageRank")
}
}
Scala 面试题
1-1）、接口&类&APP方法
trait Logger {
def log(msg: String): Unit = {
println("log:" + msg)
}
}
class ConcreteLogger extends Logger {
def concreatelog(): Unit = {
log("hello scala")
}
}
object UserTrait extends App {
val log = new ConcreteLogger
log.concreatelog()
}