The content of this article comes from the case sharing of TOP100summit Twitter technical lead for Heron Maosong Fu in 2016.
Editor: Cynthia
Maosong Fu：Technical Lead for Heron at Twitter
Introduction : The scale of data that people need to process and the speed of response to results are growing faster and faster, but Moore's Law is gradually failing, and system designers can no longer obtain huge performance improvements simply by upgrading hardware. At this time, we hope that the past single-machine tasks can be divided into many computers for parallel processing. We need distributed systems, from distributed management of resources, distributed message distribution, distributed computing to distributed storage, and more. Large-scale real-time computing has increasingly become a more and more worthy of attention due to the consideration of low latency, high throughput, and accurate calculation. To this end, Twitter has developed a new generation of real-time computing platform Heron, to address these problems.
1. Questions
To run real-time computing tasks on a large scale, the primary question is: how to lower the threshold for developing real-time computing tasks, so that developing programs that can process massive amounts of data in real time is as simple as developing single-threaded programs. Previously, we used our self-developed distributed real-time computing framework Storm (open sourced in 2011 and became the most popular real-time computing framework in the industry) to solve this problem well.
But as Twitter's data scale continues to grow, we encounter new demands and challenges, including: billions of events per minute; massive processing with sub-second latency and predictable behavior; High data accuracy; good resilience against temporary traffic spikes and pipeline congestion; easy to debug; easy to deploy in shared infrastructure.
How to process these massive amounts of data with low latency and high throughput while meeting the above requirements is a major challenge.
To this end, we have considered the following options:
● Improved Storm
● Use popular open source or commercial solutions in the industry
● Redesign and implement a real-time computing framework according to requirements
The costs of these three schemes are rising in turn, so our attempts are also from front to back.
First of all, we have made a lot of improvements to Storm, mainly for scalability and stability. The results include: enabling a single Storm cluster to accommodate 5 times as many machines as before. But if you want to continue to improve, you need to change the underlying design and implementation, and the cost is very high, no less than redesigning and implementing a real-time computing framework.
We also examined popular open source solutions and commercial solutions in the industry. The most popular in the industry at the time was Storm. And other frameworks do not meet our needs in terms of scalability, throughput and latency very well. Moreover, other systems are not compatible with Storm's API, and all real-time computing tasks need to be rewritten, which will greatly increase the migration cost. The industry also does not have a very good commercial solution for the huge real-time traffic of Twitter.
Finally, we decided to redesign and implement Heron, a real-time computing framework compatible with Storm API, based on previous experience and current needs.
2. The practical process
The following are the main design goals of Heron:
● Resource isolation – each node and computing unit in a real-time computing task should be able to use and only use those resources that are allocated to them. This enables Heron to also guarantee resource allocation and resource isolation on a shared infrastructure.
● Compatibility – Heron is fully compatible with Apache Storm's API and data model, reducing migration costs between the two systems.
● Scenario Guarantee – Heron supports at-most-once, at-least-once, exactly-once and other scenarios. And under various semantics, different trade-offs can also be achieved through different configurations. For example, in the exact-once scenario, low overhead but long recovery time, high overhead but short recovery time, and mixed mode can be selected through configuration.
● Performance – Many Heron design choices allow Heron to achieve higher throughput and lower latency than Storm, while also providing enhanced configurability to fine-tune possible latency/throughput tradeoffs.
● Efficiency – Heron was built to achieve all of the above goals with minimal resource usage.
● Provide new functionality, such as backpressure mechanisms – In a distributed system such as Heron, there is no guarantee that all system components will execute at the same speed. Heron has built-in backpressure mechanisms to ensure that the topology can adapt in the event of slow components.
The overall architecture of Heron is shown in Figure 1 and Figure 2. Users can use the Storm API to implement topologies and submit them to the resource scheduler. The resource scheduler splits a topology into multiple containers to run according to the Packing Algorithm: one of the containers runs the Topology master and is responsible for managing the topology; each of the remaining containers runs, and a stream manager is responsible for Data routing, a metrics manager to collect and report various metrics; multiple Heron instances (running user-defined spout/bolt code) processes; and other daemons. The resource scheduler automatically schedules each container according to the available resources of the cluster. Additionally, we use Zookeeper to synchronize Topology metadata.
Figure 1: Heron Architecture
Figure 2: Topology Architecture
2.1 Topology runtime components
The Heron Topology runtime includes the following components:
– Topology Master
– Stream Manager
– Heron Instance
– Metrics Manager
Topology Master
The Topology Master(TM) manages the entire lifecycle of a topology, from submission until its eventual kill. When Heron deploys a topology, it starts a TM and multiple containers. This TM creates a unique temporary ZooKeeper node so that it can be discovered by other containers; at the same time, the uniqueness of this node also ensures that the topology has only this TM. This TM is also responsible for constructing the metadata of the topology and passing it to the different components.
Stream Manager
The Stream Manager (SM) manages the routing of tuples between components. Each Heron instance in a topology is connected to its local SM, and all SMs in a given topology are connected to each other to form a network. Figure 3 is an illustration of the SM network:
Figure 3 SM network diagram
In addition to being a routing engine for data flows, the SM is also responsible for implementing backpressure mechanisms in the topology when needed. Figure 4 is an illustration of back pressure:
Figure 4 Back pressure diagram
In the image above, it is assumed that bolt B3 (in container A) has all inputs from spout S1. B3 runs slower than other components. The result: container A's SM would reject input from containers C and D, because that would cause the buffers of those containers to overflow, causing the process to crash.
In this case, Heron's backpressure mechanism works. The SM in container A sends a message to all other SMs to notify the data source to reduce data traffic. Figure 5.
Figure 5 Heron's back pressure mechanism works
Once the backward bolt (B3) returns to normal, the SM of container A will notify other SMs, and the flow routing of this topology will return to normal.
Heron Instance
A Heron Instance (HI) is a process that handles independent spout or bolt tasks, supporting simple debugging and analysis. Currently, Heron only supports Java, so all HIs are JVM processes, but that will change in the future.
Metrics Manager
Each topology runs a Metrics Manager (MM) for collecting and exporting operational parameter metrics for all components in a container. Then send these running parameter metrics to Topology Master and or other metrics collectors, such as Scribe, Graphite, etc.
We also allow users to implement their own Metrics Sink to enable Heron to support other systems.
2.2 Heron features:
● Off the shelf scheduler
We abstract the popular resource schedulers in the industry, so that Heron can easily run on various existing resource scheduling frameworks, such as: Mesos, YARN, Docker and so on. In this way, the existing resource scheduling framework can be used without deploying a cluster for Heron, which greatly reduces the cost of deployment and maintenance.
● Handling spikes and congestion
Heron has a back pressure mechanism, that is, it dynamically adjusts the data flow in a topology during execution, so that when the upstream and downstream processing speeds of the tasks are inconsistent, it can still run well without losing data to ensure the correctness of the analysis results. This is useful during flow peaks and blocked pipes.
● Easy debugging
Each task is isolated at the process level, making it easy to understand behavior, performance, and file configuration. In addition, Heron has a built-in UI as shown in Figure 6, which can automatically display the corresponding parameters, which is convenient for troubleshooting problems quickly and effectively, and displays the logical plan, physical plan and various real-time operating parameter indicators.
Picture 6: Heron UI
● Compatibility with Storm
Heron provides full Storm compatibility, eliminating the need to spend too much time and resources researching new systems. Plus, you can run existing Storm topologies in Heron without changing any code, allowing for easy migration.
● Scalability and latency
Heron can handle large-scale topologies with high throughput and low latency requirements. Furthermore, the system can handle a large number of topologies.
2.3 Heron performance
Comparing Heron and Storm, the sample stream is 150,000 words, as shown in Figure 7:
图7. Throughput with acks enabled
图8. Latency with acks enabled
As shown in Figure 7, the throughputs of Heron and Storm show a linear growth trend. However, Heron throughput is 10–14 times higher than Storm in all experiments. Also in terms of end-to-end latency, as shown in Figure 8, both are increasing, but Heron latency is 5–15 times lower than Storm.
Beyond that, Twitter already runs topologies in the hundreds of machines, many of which implement resource processing that generates millions of events per second without issue. With Heron, many topologies can achieve sub-second latency for cluster data per second. In these cases, Heron can achieve its goals with lower resource consumption than Storm.
2.4 Heron at Twitter
At Twitter, Heron serves as the primary streaming system, running millions of development and production topologies. Due to Heron's efficient use of resources, after migrating all of Twitter's topologies, the overall hardware was reduced by a factor of 3, resulting in a significant improvement in the efficiency of Twitter's infrastructure setup.
For more TOP100 case information and schedule, please visit [Official Website]. Including product, team, architecture, operation and maintenance, big data, artificial intelligence and other technical special sessions, 4 days will focus on sharing the 100 most worthwhile research and development case practices at home and abroad in 2017. A total of 10 single-day free trial tickets for the opening ceremony will be sent out on this platform, and the number is limited on a first-come, first-served basis. Free trial ticket application entrance