Storm Akka Finagle对比及使用场景分析

I was recently working a bit with Twitter’s Storm, and it got me wondering, how does it compare to another high-performance, concurrent-data-processing framework, Akka. What’s Akka and Storm? Let’s start with a short description of both systems. Storm is a distributed, real-time computation system. On a Storm cluster, you execute topologies, which process streams of tuples (data). Each topology is a graph consisting of spouts (which produce tuples) and bolts (which transform tuples). Storm takes care of cluster communication, fail-over and distributing topologies across cluster nodes.	译者信息 最近一段时间工作很多都在和Twitter’s Storm打交道，这使我想知道，它和另一个高性能、并发数据处理的框架Akka对比有什么优缺点呢？ 关于Akka和Storm？ 让我们简单的描述一下两个系统。Storm是一个分布式实时计算系统。在一个Storm集群中，执行拓扑结构，在这个拓扑中进行元组（数据）流进程。每一 个拓扑是一个由喷嘴（他产生元组）和螺栓（他负责转化元组）组成。Storm负责集群通讯，容错和通过集群节点分配拓扑。
Akka is a toolkit for building distributed, concurrent, fault-tolerant applications. In an Akka application, the basic construct is an actor; actors process messages asynchronously, and each actor instance is guaranteed to be run using at most one thread at a time, making concurrency much easier. Actors can also be deployed remotely. There’s a clustering module coming, which will handle automatic fail-over and distribution of actors across cluster nodes. Both systems scale very well and can handle large amounts of data. But when to use one, and when to use the other? There’s another good blog post on the subject, but I wanted to take the comparison a bit further: let’s see how elementary constructs in Storm compare to elementary constructs in Akka.	译者信息 Akka 是一个构建分布式、并发、容错应用程序的工具集。在Akka应用中，基本结构是一个物件；物件异步的处理消息，并且保证每一个物件在某一个时刻最多运行在一个线程中，这是的同步更加容易。物件也可以远程部署。这儿是一个集群模块，它可以自动处理实现容错和通过集群节点进行物件的分配。 两个系统都大型架构可以处理超大数量的数据。但是在使用时候我们怎么选择呢？ 这儿有一个很好的博文论述了这个主题，但是我想跟进一步对他们做个比较：让我们看看两者最基本的结构有什么不同？
Comparing the basics Firstly, the basic unit of data in Storm is a tuple. A tuple can have any number of elements, and each tuple element can be any object, as long as there’s a serializer for it. In Akka, the basic unit is a message, which can be any object, but it should be serializable as well (for sending it to remote actors). So here the concepts are almost equivalent. Let’s take a look at the basic unit of computation. In Storm, we have components: bolts and sprouts. A bolt can be any piece of code, which does arbitrary processing on the incoming tuples. It can also store some mutable data, e.g. to accumulate results. Moreover, bolts run in a single thread, so unless you start additional threads in your bolts, you don’t have to worry about concurrent access to the bolt’s data. This is very similar to an actor, isn’t it? Hence a Storm bolt/sprout corresponds to an Akka actor. How do these two compare in detail? Actors can receive arbitrary messages; bolts can receive arbitrary tuples. Both are expected to do some processing basing on the data received. Both have internal state, which is private and protected from concurrent thread access.	译者信息 基本比较 首先，Storm最基本的数据部分是一个元组。一个元组有人以数量的元素构成，每一个元组元素可以是任意的对象，并且实现了序列化。而Akka的最基本单 元是消息，消息也可以是任意的对象，同样他应该可被序列化（为了把它送到远程的物件）。所以在概念上两者基本上是一样的。 让我们看看基本的计算单元。在Storm中，拥有组件：螺栓和喷嘴。一个螺栓可以是任意块的代码，可在随后的元组中处理任意的进程。他也可以存储一些动态 的数据。例如，累加的结果。同时，螺栓运行在单线程，所以除非你要在你的螺栓中开启一个新的线程，你不用担心并行处理数据的问题。这和物件非常像,不是 吗？因此，一个Storm螺栓/喷嘴相当于一个Akka物件。那两者的细节上有上有和异同呢？ 物件可以接收任意消息；螺栓可以接收任意的元组。两者都被期望在接收到的数据上做一些处理。两者都有内部状态，包括对并发线程处理的私有的和受保护态
Actors & bolts: differences One crucial difference is how actors and bolts communicate. An actor can send a message to any other actor, as long as it has theActorRef(and if not, an actor can be looked up by-name). It can also send back a reply to the sender of the message that is being handled. Storm, on the other hand is one-way. You cannot send back messages; you also can’t send messages to arbitrary bolts. You can also send a tuple to a named channel (stream), which will cause the tuple (message) to be broadcast to all listeners, defined in the topology. (Bolts also ack messages, which is also a form of communication, to the ackers.) In Storm, multiple copies of a bolt’s/sprout’s code can be run in parallel (depending on the parallelism setting). So this corresponds to a set of (potentially remote) actors, with a load-balancer actor in front of them; a concept well-known from Akka’s routing. There are a couple of choices on how tuples are routed to bolt instances in Storm (random, consistent hashing on a field), and this roughly corresponds to the various router options in Akka (round robin, consistent hashing on the message).	译者信息 物件和螺栓的不同 一个显著的不同是两者的通讯方式。一个物件可以传递消息给任意的物件，只要有物件的引用（如果没有，一个物件可以通过名称查询出）。物件也可以给消息发送 者返回一个回复，说明消息已被处理。Storm处理的过程则是单向的。他不能返回消息。你也不能给任意的螺栓传递消息。你可以把一个元组放入一个命名通道 （流），这个通道可以把螺栓发出的消息在所在通道进行广播，在拓扑中定义（螺栓也确认消息，这也是一种通讯的方式，对确认者） 在Storm中，多份螺栓/喷嘴的代码拷贝可以并行执行（取决于并行设置）。这相当于一系列（可能是远程的）物件，但是在前端有一个负载均衡的物件；Akka中的概念叫路由。关于元组如何被传递给螺栓实例有很多中的算法，包括随机、一个域的相容哈希等。这和Akka中的多种路由选择类似（循环、消息相容哈希）
There’s also a difference in the “weight” of a bolt and an actor. In Akka, it is normal to have lots of actors (up to millions). In Storm, the expected number of bolts is significantly smaller; this isn’t in any case a downside of Storm, but rather a design decision. Also, Akka actors typically share threads, while each bolt instance tends to have a dedicated thread. Other features Storm also has one crucial feature which isn’t implemented in Akka out-of-the-box: guaranteed message delivery. Storm tracks the whole tree of tuples that originate from any tuple produced by a sprout. If all tuples aren’t acknowledged, the tuple will be replayed. Also the cluster management of Storm is more advanced (automatic fail-over, automatic balancing of workers across the cluster; based on Zookeeper); however the upcoming Akka clustering module should address that.	译者信息 在螺栓和物件在构成权重上有有所不同。在Akka，并列地存在这很多物件（可以到数百万）。在Storm中，预期的螺栓数量是很小的。没有任何的理由限制这个数量，这实际上是由设计初衷导致的。同时，Akka物件通常都是共享线程的。而螺栓实例却是独享线程。 其他特性 Storm 也有一个重要的特性，有在Akka中所没有的：保证消息传递。Storm遍历整个元组数，这个数的根节点都是嘴发出的元组。如果所有的元组还没有被确认，元组将被重新遍历。 同时，Storm的集群管理也是更先进（自动容错，集群中自动负载均衡；基于Zookeeper）。一个好消息是Akka的集群模块将会增加这些特性。
Finally, the layout of the communication in Storm – the topology – is static and defined upfront. In Akka, the communication patterns can change over time and can be totally dynamic; actors can send messages to any other actors, or can even send addresses (ActorRefs). So overall, Storm implements a specific range of usages very well, while Akka is more of a general-purpose toolkit. It would be possible to build a Storm-like system on top of Akka, but not the other way round (at least it would be very hard). Adam	译者信息 最后，Storm的层状通讯、拓扑是静态的，并且是事先定义的。Akka通讯模式可以通过时间转换，并且可以完全动态配置。物件可以传递消息给任何其他物件，也可以送抵到地址（物件引用）。 总而言之，Storm适用于特定范围的用户，而Akka是一个更加通用的工具集。基于Akka可以构建类似Storm的系统，但是相反做法是不可行的（至少可以可以说那将非常的艰难） 亚当     Storm是一个分布式实时计算系统。就像Hadoop提供一组通用原语来进行批量处理(batch processing)一样，Storm也提供了一组通用原语来进行实时计算(realtime computation)。Storm非常简单，能用于任意编程语言，被很多大的公司采用，并且使用过程中乐趣多多。     本教程中，你会学习如何创建Storm拓扑(topologies)，以及如何部署它们到Storm集群中。Java是我们使用的主要语言，但是一些例子会使用到Python来展示Storm的多语言能力。     Storm集群组件(Components of a Storm cluster)     Storm集群看起来类似于Hadoop集群。然而，在Hadoop上你运行的是”MapReduce jobs”，在Storm上你运行的是” topologies”。”Jobs”和”topologies”本身就非常的不同，其中一个主要的一个区别是，MapReduce的任务最终会结束，而 拓扑则永远在处理消息(直到你干掉它)。     torm集群上有两种类型的节点：主节点(master node)和工作节点(worker node)。主节点运行一个称为Nimbus的守护进程，类似于Hadoop的JobTracker。Nimbus负责分发代码到集群中，分配任务到机 器，并且监控失败。     每一个工作节点运行一个称为Supervisor的守护进程。supervisor监听分配给这台机器的工作，并且基于Nimbus分配情况在必要时启动和停止工作进程。每一个工作进程执行一个拓扑的子集合；一个运行中的拓扑由横跨多个机器的多个工作进程组成。