Editor's Note
Mesos and Yarn are very good resource scheduling framework, the community there are a lot of people and the use of analysis of difference between the two scenarios. Before InfoQ also talked to the two questions of the relationship . The industry is using more Mesos, this article is to explain why the author chose to use Yarn instead Mesos, and describes how to develop distributed applications based on Yarn. This article first appeared to wish William's blog , authorized reprint published by InfoQ.
Foreword
Mesos In fact, I am not very familiar with, so some content may be biased, with personal preferences. We still need to have their ability to identify.
Yarn Mesos and are great, are programmable frame. A hardware, not programming, is dead, it can be programmed once live, you can toss all kinds, a variety of whimsy can achieve the same, a software, as long as the programmable, basic will to live, easy formation of ecological.
Yarn VS Mesos
When I first talk about doing container scheduling engine, why choose Yarn instead of Mesos.
Deployability
First note, explored here is the deployment of Yarn or Mesos cluster, which does not involve the application. In addition to the JDK Yarn dependence, no dependence on an operating system, able to run substantially put up. Mesos because C / C ++ development, installation deployment may have library dependencies. This I do not know whether we look at the weight, anyway, I see very heavy. Software should be lower down on Run. So 12 years when I myself developed a Java Services Framework, run after the completion of the development of a main method on the line. Let applications include container, rather than make Tomcat application thrown into these containers, too complex and not intuitive.
Secondary development
Yarn for Java / Scala engineer, just a Jar package, similar index development package Lucene, you can put it to introduce the project, do any package you want. This is one.
Second, Yarn provides a lot of expansion interface, many implementations are pluggable. Alternatively, in the XML configuration, it can easily replace it with your implementation off the original implementation, not much invasive, so even if the future Yarn upgrade, there will not be a big problem.
In comparison, Mesos more like a ready made product, can be deployed directly, but not friendly for secondary development.
Ecological advantages
Yarn Hadoop was born in the "initiator" big data project, it has inherent advantages in the field of big data.
Distributed storage system is the underlying natural HDFS, stable and efficient.
On top of the seat supporting the carry large data field Spark, MR et al., Tried and tested.
Strong community, recently released version is also significantly faster, to support long task has become increasingly outstanding.
Long mission support
Turning to support long task (long running services), it was previously thought to support offline Yarn short task, it may have limited support for long missions. In fact, need not worry, for example, is now based on Spark Streaming is 7x24 hours of running, running up and also lacks the problem. In general, to support long task, consider the following several points:
Fault tolerance, mainly AM fault tolerance.
Yarn Security, if turned on the security mechanisms, such as token expiration time is also needed attention.
Log collection to the cluster.
There is a resource isolation and priority. If the resource isolation too bad to do, the task will have an impact long time.
We are interested can refer to Jira . I think this Jira 13 years began, indicating that this matter has long taken seriously again. Here are several points mentioned in our team to do the next explanation.
Fault tolerance
Yarn their availability. Currently Yarn of Master has been achieved HA.
AM fault tolerance, I do keep containers across attempt option from the 2.4 version (see the source code, or it may have been supported by earlier versions) has been supported. What does that mean? If AM is hung up in the process of restarting the Yarn AM, all containers will be kept by the management of AM without being killed. Unless Yarn several attempts AM no way to restart it (default twice). This shows that from the bottom of the scheduling point of view, has done very well.
Log collection to the cluster
Log collection in version 2.6 is already running side edge collected.
Resource isolation
Resource isolation, then, Yarn do not, currently in force is the memory, other aspects have always wanted to support, but has been limited. It is also estimated that many people choose Mesos last reason. But now this advantage Mesos is already gone, because Docker containers on the resource isolation has done well enough. Yarn and Docker an integrated, complemented.
summary
Mesos and Yarn are very good scheduling framework, each has its advantages and disadvantages, flexible scheduling, unified resource management platform is a trend in the future, similar to the management of this resource scheduling framework will certainly be popular.
Some common misconceptions
Born out of Hadoop, inherited his aura and ecology, but it will also bring some confusion to which the first is to Hadoop aura has been covered, and because of the inherent inertia, we will take for granted that just Yarn in Hadoop a component of the Yarn thought someone would come up with it alone?
However, as I have before a curriculum, he has repeatedly stressed, Hadoop is a software collection that contains distributed storage, resource management, scheduling, calculation of three parts of the frame. There is no necessary relationship between them, is open to independent. Yarn is a resource management and scheduling engine, one design goal is to begin general, not just run MR. Now based service has been very much on top of Yarn, typical of such Spark.
There is another misunderstanding, MR currently offline batch basically be synonymous with this time mistaken Yarn is only suitable for off-line batch scheduling tasks. In actual fact, I have given in the above analysis, Yarn is entirely possible to ensure stable long task of reliable operation.
How to develop distributed applications based on Yarn
This article will not teach you how to use specific Yarn of API, but if you want to know Yarn of API, but also feel that the official document is too brief, here I'd be given two suggestions:
Examples can be found in the use of codes Spark Yarn relevant code. Counted on is a very streamlined Yarn of adaptor.
Buy a Yarn-related book, to understand its architecture is also designed to help you understand its API.
The following content will explore the following two topics:
Yarn-based development of distributed application needs to do some preparation work
Yarn based on some basic ideas to develop container dispatch system
Yarn-based development of distributed application needs to do some preparation work
Certainly can not roll up its sleeves and began to dry. Before starting hands, we need to know what things?
Too underlying API Yarn native, too complicated
If you want to stay in the development of Yarn application, then the Yarn once the API package is necessary. Yarn For flexibility, or to be able to meet the needs of most developers, the underlying interaction of the API becomes more original. Naturally caused great development effort. This is not a person I think, now the Apache Twill, and when they develop Hulu Adaptor that layer, they are actually in order to solve this problem. Twill why I did not use it, the first document is too little, and the second is a bit more complicated, I do not need such a complicated thing. I think, Twill its development so versatile, really good as well write about the document.
The best is to develop a solution to a class of problems Framework
Yarn is just a low-level resource management and scheduling engine. You generally need to be based on the development of a Framework solving specific problems. To Spark example, he is to solve some of the problems associated with distributed computing. The container with the scheduler I developed, in fact, is to address the deployment of dynamic Web applications. Above them, it is your application. For example, you have to log statistics, as long as you develop an Application on the Spark. For example, you want to provide a recommendation system, so long as you use a lower container packaging container can be scheduled deployment scheduler.
So general, distributed applications based on Yarn should be consistent with such a level:
Yarn -> Adapter -> Framework -> Application
Adapter is what I first said it, you self a package of Yarn of API. Framework programming framework is to solve a class of problems, Application is what you really want to solve the systems business. Through this decoupling, at all levels as long as the focus on their core function point can be.
Ensure that your upper Framework / Application can be transferred
Spark is a typical, he can run on Mesos, you can also run on Yarn, can also run on itself (Standalone), real-time, soak in Yarn, and run Standalone mode, have a lot of. Thanks Spark itself is not dependent on the underlying resource management scheduling engine.
Actually, this is what I said above two benefits, because Adaptor, upper Framework can not tie up on a resource scheduling engine. The Framework can make Applicaiton without paying attention to the underlying scheduling matter, as long as you can focus on business.
In addition, your pains development Framework, you naturally hope that it will run on more platforms, has met the needs of more people, right.
Yarn based on some basic ideas to develop container dispatch system
First, we need to understand two concepts:
Dumb application. The so-called dumb application refers can not interact directly and distributed systems, distributed systems can be controlled only through the container life cycle, such as closed or open applications. A typical example MySQL, Nginx and so on the basis of these applications. They generally have their own unique way to interact, such as the command line or socket protocol, or HTTP protocol.
Associated components. Because there has been dumb applications, distributed systems and to be able to interact with these applications, you need to have an agent. And this agent and the agent's dumb application, have the same life cycle. A typical example, after a service has been shut down, the event is informed of a distributed system, the distributed system sends the event to the associated Nginx assembly, associated components into Nginx instruction can be recognized, the service will be stopped from ProxyBackend list of Nginx excluded.
In the container dispatch system, if Yarn of NodeManager directly to manage Docker Yarn will need to do to support itself, I think this is wrong. Yarn duty is to do resource management, allocation, scheduling can be, and does not require a specific coupling techniques, after all Yarn is a universal resource scheduling management framework.
That based on the above theory, we based Yarn, develop a framework that would be typical master-slave structure (which is Yarn decision). slaves of this framework are actually Docker's companion object. master may be achieved indirectly by administration of these containers Slave.
We briefly describe their processes under:
Users submit Application, application resources;
Yarn start the master Framework;
Yarn start of slave Framework;
master slave connection, and sending the heartbeat, so that the slave master knows the slave server starts Docker condition, the slave with the activated one-docker container;
Container slave regular monitoring;
slave found container crash, slave automatically exit, Yarn notified recover resources;
found master node fails, the node issues a new request to restart the slave on another server, step 2 is repeated from the beginning.
There is also a problem, if the slave is normally kill, incidentally, can also be turned off by the Container JVM ShudownHook. But if the slave is abnormal crash or kill -9 lost, then it may lead to a resource leak. This information is currently reported by the master to the cluster management platform, which will be cleaned regularly. You can also store this information, such as MySQL or put Redis, and then start the background cleanup task can be.
Knowing this idea, the specific implementation is simplified, is to develop a master-slave program can Yarn-based, then the corresponding slave to manage Docker containers, including the acceptance of new instructions. master container provides management interface display information, operating status can be.
Of course, you can then develop a Framework B specialized and Nginx interaction, such as the above systems do such a node changes, notify the master B, and master B via their associated components Slave complete the update Nginx, enabling backend and automatic change notification service.
It now appears that this is not a perfect concept covers interaction between the application of it?