Instances of any type must go through the serialization/deserialization/deserialize process when they are passed between machines of independent systems at both ends as a message. Suppose the following scenario: There are two connected servers in a network, and they each deploy an independent akka system. All messages have to be serialized/deserialized if we need to exchange messages between the akka systems of these two servers. The default serialization of akka system for user-defined message types is performed in the way of java-object serialization. As we mentioned last time: because java-object-serialization will write the type information, instance value, and other type description information contained in a java-object into the serialized result, it will occupy a large space. The efficiency of data transmission is relatively low. Protobuf is in binary format and basically only includes instance values, so the data transmission efficiency is high. Below we will introduce how to use protobuf serialization in the akka system. Using a custom serialization method in akka involves the following steps:
1. IDL definition of the message type in the .proto file
2. Compile the IDL file with ScalaPB and generate the scala source code. These source codes include the types of messages involved and their operation methods
3. Import the generated classes in the akka program module, and then call these types and methods directly
4. Write serialization method according to akka requirements
5. Define akka's default serializer in the actor.serializers section of akka's .conf file
The program structure is described in the following build.sbt file:
lazy val commonSettings = Seq(
name := "AkkaProtobufDemo",
version := "1.0",
scalaVersion := "2.12.6",
)
lazy val local = (project in file("."))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11",
"com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"
),
name := "akka-protobuf-demo"
)
lazy val remote = (project in file("remote"))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11"
),
name := "remote-system"
).dependsOn(local)
PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
local and remote are two separate projects. We will deploy the akka system in these two projects respectively. Note the scalapb.runtime in the dependencies. PB.targets specifies the path to generate the source code. We also need to specify the scalaPB plugin in project/scalapb.sbt:
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.7.1"
我们首先在.proto文件里定义消息：
syntax = "proto3";
// Brought in from scalapb-runtime
import "scalapb/scalapb.proto";
import "google/protobuf/wrappers.proto";
package learn.proto;
message Added {
int32 nbr1 = 1;
int32 nbr2 = 2;
}
message Subtracted {
int32 nbr1 = 1;
int32 nbr2 = 2;
}
message AddedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
}
message SubtractedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
}
现在我们先在remote项目里定义一个Calculator actor：
package akka.protobuf.calculator
import akka.actor._
import com.typesafe.config.ConfigFactory
import learn.proto.messages._
class Calculator extends Actor with ActorLogging {
override def receive: Receive = {
case Added(a,b) =>
log.info("Calculating %d + %d".format(a, b))
sender() ! AddedResult(a,b,a+b)
case Subtracted(a,b) =>
log.info("Calculating %d - %d".format(a, b))
sender() ! SubtractedResult(a,b,a-b)
}
}
object Calculator {
def props = Props(new Calculator)
}
object CalculatorStarter extends App {
val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2552")
.withFallback(ConfigFactory.load())
val calcSystem = ActorSystem("calcSystem",config)
calcSystem.actorOf(Calculator.props,"calculator")
println("press any key to end program ...")
scala.io.StdIn.readLine()
calcSystem.terminate()
}
运行CalculatorStarter产生一个calculator actor: akka.tcp://[email protected]:2552/user/calculator
下面我们在local项目里从端口2551上部署另一个akka系统，然后调用端口2552上部署akka系统的calculator actor：
package akka.protobuf.calcservice
import akka.actor._
import learn.proto.messages._
import scala.concurrent.duration._
class CalcRunner(path: String) extends Actor with ActorLogging {
sendIdentifyRequest()
def sendIdentifyRequest(): Unit = {
context.actorSelection(path) ! Identify(path)
import context.dispatcher
context.system.scheduler.scheduleOnce(3.seconds, self, ReceiveTimeout)
}
def receive = identifying
def identifying : Receive = {
case ActorIdentity(calcPath,Some(calcRef)) if (path.equals(calcPath)) =>
log.info("Remote calculator started!")
context.watch(calcRef)
context.become(calculating(calcRef))
case ActorIdentity(_,None) =>
log.info("Remote calculator not found!")
case ReceiveTimeout =>
sendIdentifyRequest()
case s @ _ =>
log.info(s"Remote calculator not ready. [$s]")
}
def calculating(calculator: ActorRef) : Receive = {
case (op : Added) => calculator ! op
case (op : Subtracted) => calculator ! op
case AddedResult(a,b,r) =>
log.info(s"$a + $b = $r")
case SubtractedResult(a,b,r) =>
log.info(s"$a - $b = $r")
case Terminated(calculator) =>
log.info("Remote calculator terminated, restarting ...")
sendIdentifyRequest()
context.become(identifying)
case ReceiveTimeout => //nothing
}
}
object CalcRunner {
def props(path: String) = Props(new CalcRunner(path))
}
这个CalcRunner是一个actor，在程序里首先通过向remote项目中的calculator-actor传送Identify消息以取得具体的ActorRef。然后用这个ActorRef与calculator-actor进行交互。这其中Identify是akka预定消息类型，其它消息都是ScalaPB从.proto文件中产生的。下面是local项目的运算程序：
package akka.protobuf.demo
import akka.actor._
import akka.util.Timeout
import com.typesafe.config.ConfigFactory
import akka.protobuf.calcservice._
import scala.concurrent.duration._
import scala.util._
import learn.proto.messages._
object Main extends App {
val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2551")
.withFallback(ConfigFactory.load())
val calcSystem = ActorSystem("calcSystem",config)
val calcPath = "akka.tcp://[email protected]:2552/user/calculator"
val calculator = calcSystem.actorOf(CalcRunner.props(calcPath),"calcRunner")
println("Calculator started ...")
import calcSystem.dispatcher
calcSystem.scheduler.schedule(1.second, 1.second) {
if (Random.nextInt(100) % 2 == 0)
calculator ! Added(Random.nextInt(100), Random.nextInt(100))
else
calculator ! Subtracted(Random.nextInt(100), Random.nextInt(100))
}
scala.io.StdIn.readLine()
}
配置文件application.conf：
akka {
actor {
provider = remote
}
remote {
netty.tcp {
hostname = "127.0.0.1"
}
}
}
先运行remote然后local。注意下面出现的提示：
[akka.serialization.Serialization(akka://calcSystem)] Using the default Java serializer for class [learn.proto.messages.Added] which is not recommended because of performance implications. Use another serializer
下面是protobuf类型的序列化方法：
package akka.protobuf.serializer
import akka.serialization.SerializerWithStringManifest
import learn.proto.messages._
class ProtobufSerializer extends SerializerWithStringManifest{
def identifier: Int = 101110116
override def manifest(o: AnyRef): String = o.getClass.getName
final val AddedManifest = classOf[Added].getName
final val SubtractedManifest = classOf[Subtracted].getName
final val AddedResultManifest = classOf[AddedResult].getName
final val SubtractedResultManifest = classOf[SubtractedResult].getName
override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef = {
println("inside fromBinary"+manifest)
manifest match {
case AddedManifest => Added.parseFrom(bytes)
case SubtractedManifest => Subtracted.parseFrom(bytes)
case AddedResultManifest => AddedResult.parseFrom(bytes)
case SubtractedResultManifest => SubtractedResult.parseFrom(bytes)
}
}
override def toBinary(o: AnyRef): Array[Byte] = {
println("inside toBinary ")
o match {
case a: Added => a.toByteArray
case s :Subtracted => s.toByteArray
case aR: AddedResult => aR.toByteArray
case sR: SubtractedResult => sR.toByteArray
}
}
}
然后我们需要在application.conf中告诉akka系统使用这些方法：
actor {
serializers {
proto = "akka.protobuf.serializer.ProtobufSerializer"
}
serialization-bindings {
"java.io.Serializable" = none
"com.google.protobuf.Message" = proto
"learn.proto.messages.Added" = proto
"learn.proto.messages.AddedResult" = proto
"learn.proto.messages.Subtracted" = proto
"learn.proto.messages.SubtractedResult" = proto
}
}
现在再重新运行：
[INFO] [04/30/2018 18:41:02.348] [calcSystem-akka.actor.default-dispatcher-2] [akka.tcp://[email protected]:2551/user/calcRunner] Remote calculator started!
inside toBinary
inside fromBinarylearn.proto.messages.AddedResult
[INFO] [04/30/2018 18:41:03.234] [calcSystem-akka.actor.default-dispatcher-4] [akka.tcp://[email protected]:2551/user/calcRunner] 18 + 38 = 56
inside toBinary
inside fromBinarylearn.proto.messages.AddedResult
[INFO] [04/30/2018 18:41:04.197] [calcSystem-akka.actor.default-dispatcher-4] [akka.tcp://[email protected]:2551/user/calcRunner] 22 + 74 = 96
系统使用了自定义的ProtobufferSerializer。
下面是本次示范的完整源代码：
project/scalapb.sbt
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.7.1"
build.sbt
lazy val commonSettings = Seq(
name := "AkkaProtobufDemo",
version := "1.0",
scalaVersion := "2.12.6",
)
lazy val local = (project in file("."))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11",
"com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"
),
name := "akka-protobuf-demo"
)
lazy val remote = (project in file("remote"))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11"
),
name := "remote-system"
).dependsOn(local)
PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
resources/application.conf
akka {
actor {
provider = remote
}
remote {
netty.tcp {
hostname = "127.0.0.1"
}
}
actor {
serializers {
proto = "akka.protobuf.serializer.ProtobufSerializer"
}
serialization-bindings {
"java.io.Serializable" = none
"com.google.protobuf.Message" = proto
"learn.proto.messages.Added" = proto
"learn.proto.messages.AddedResult" = proto
"learn.proto.messages.Subtracted" = proto
"learn.proto.messages.SubtractedResult" = proto
}
}
}
main/protobuf/messages.proto
syntax = "proto3";
// Brought in from scalapb-runtime
import "scalapb/scalapb.proto";
import "google/protobuf/wrappers.proto";
package learn.proto;
message Added {
int32 nbr1 = 1;
int32 nbr2 = 2;
}
message Subtracted {
int32 nbr1 = 1;
int32 nbr2 = 2;
}
message AddedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
}
message SubtractedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
}
remote/Calculator.scala
package akka.protobuf.calculator
import akka.actor._
import com.typesafe.config.ConfigFactory
import learn.proto.messages._
class Calculator extends Actor with ActorLogging {
override def receive: Receive = {
case Added(a,b) =>
log.info("Calculating %d + %d".format(a, b))
sender() ! AddedResult(a,b,a+b)
case Subtracted(a,b) =>
log.info("Calculating %d - %d".format(a, b))
sender() ! SubtractedResult(a,b,a-b)
}
}
object Calculator {
def props = Props(new Calculator)
}
object CalculatorStarter extends App {
val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2552")
.withFallback(ConfigFactory.load())
val calcSystem = ActorSystem("calcSystem",config)
calcSystem.actorOf(Calculator.props,"calculator")
println("press any key to end program ...")
scala.io.StdIn.readLine()
calcSystem.terminate()
}
CalcService.scala
package akka.protobuf.calcservice
import akka.actor._
import learn.proto.messages._
import scala.concurrent.duration._
class CalcRunner(path: String) extends Actor with ActorLogging {
sendIdentifyRequest()
def sendIdentifyRequest(): Unit = {
context.actorSelection(path) ! Identify(path)
import context.dispatcher
context.system.scheduler.scheduleOnce(3.seconds, self, ReceiveTimeout)
}
def receive = identifying
def identifying : Receive = {
case ActorIdentity(calcPath,Some(calcRef)) if (path.equals(calcPath)) =>
log.info("Remote calculator started!")
context.watch(calcRef)
context.become(calculating(calcRef))
case ActorIdentity(_,None) =>
log.info("Remote calculator not found!")
case ReceiveTimeout =>
sendIdentifyRequest()
case s @ _ =>
log.info(s"Remote calculator not ready. [$s]")
}
def calculating(calculator: ActorRef) : Receive = {
case (op : Added) => calculator ! op
case (op : Subtracted) => calculator ! op
case AddedResult(a,b,r) =>
log.info(s"$a + $b = $r")
case SubtractedResult(a,b,r) =>
log.info(s"$a - $b = $r")
case Terminated(calculator) =>
log.info("Remote calculator terminated, restarting ...")
sendIdentifyRequest()
context.become(identifying)
case ReceiveTimeout => //nothing
}
}
object CalcRunner {
def props(path: String) = Props(new CalcRunner(path))
}
Main.scala
package akka.protobuf.demo
import akka.actor._
import akka.util.Timeout
import com.typesafe.config.ConfigFactory
import akka.protobuf.calcservice._
import scala.concurrent.duration._
import scala.util._
import learn.proto.messages._
object Main extends App {
val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2551")
.withFallback(ConfigFactory.load())
val calcSystem = ActorSystem("calcSystem",config)
val calcPath = "akka.tcp://[email protected]:2552/user/calculator"
val calculator = calcSystem.actorOf(CalcRunner.props(calcPath),"calcRunner")
println("Calculator started ...")
import calcSystem.dispatcher
calcSystem.scheduler.schedule(1.second, 1.second) {
if (Random.nextInt(100) % 2 == 0)
calculator ! Added(Random.nextInt(100), Random.nextInt(100))
else
calculator ! Subtracted(Random.nextInt(100), Random.nextInt(100))
}
scala.io.StdIn.readLine()
}
ProtobufferSerializer.scala
package akka.protobuf.serializer
import akka.serialization.SerializerWithStringManifest
import learn.proto.messages._
class ProtobufSerializer extends SerializerWithStringManifest{
def identifier: Int = 101110116
override def manifest(o: AnyRef): String = o.getClass.getName
final val AddedManifest = classOf[Added].getName
final val SubtractedManifest = classOf[Subtracted].getName
final val AddedResultManifest = classOf[AddedResult].getName
final val SubtractedResultManifest = classOf[SubtractedResult].getName
override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef = {
println("inside fromBinary"+manifest)
manifest match {
case AddedManifest => Added.parseFrom(bytes)
case SubtractedManifest => Subtracted.parseFrom(bytes)
case AddedResultManifest => AddedResult.parseFrom(bytes)
case SubtractedResultManifest => SubtractedResult.parseFrom(bytes)
}
}
override def toBinary(o: AnyRef): Array[Byte] = {
println("inside toBinary ")
o match {
case a: Added => a.toByteArray
case s :Subtracted => s.toByteArray
case aR: AddedResult => aR.toByteArray
case sR: SubtractedResult => sR.toByteArray
}
}
}