After Hadoop 2.0, the management of cluster resources was extracted from the JobTracker of MapReduce v1 and implemented in YARN. Although YARN supports a variety of different computing frameworks, it still does not solve the problem of elastic scaling of cluster resources. This article introduces a new project - Myriad, which combines the advantages of both YARN and Mesos, not only making YARN more flexible to run and use, but also making the expansion of the entire data center easier.
This is a story about two clusters. The first is an Apache Hadoop cluster, where resources are completely isolated from Hadoop and processes. Another cluster is a description of all resources that are not part of the Hadoop cluster. Distinguishing the two clusters in this way is because Hadoop manages its own resources through Apache YARN (Yet Another Resource Negotiator). For Hadoop, these resources are often underutilized when no big data tasks are queued. When a big data task runs, these resources are quickly exhausted and more resources are requested. This is quite difficult for the first cluster.
Although Hadoop intentionally intends to remove data barriers, while some barriers are removed, other types of barriers are created in the same places. As another technical solution, Apache Mesos also intends to remove these barriers. But Mesos is often used to manage a second type of cluster that includes all resources except Hadoop tasks.
The differences between Mesos and YARN introduced earlier are just the beginning of the story. Just as they are not compatible, they often compete with each other. My story, however, is that they work together.
Introduction to Mesos and YARN
The main difference between Mesos and YARN revolves around the design of priorities and the way tasks are scheduled. Mesos was born at UC Berkeley in 2007 and has been continuously consolidated under the commercialization of companies such as Twitter and Airbnb. It was originally designed as a scalable global resource manager for the entire data center. YARN out of the need to manage the scale of Hadoop. Before the advent of YARN, resource management (functionality) was integrated in the Hadoop MapReduce V1 architecture, which was removed (moved to YARN) in order to facilitate the expansion of MapReduce. MapReduce's Job Tracker cannot efficiently schedule MapReduce tasks on more than a thousand machines. YARN was created in the next-generation Hadoop lifecycle, mainly around resource scaling.
Mesos scheduling
Mesos determines which resources are available, and returns allocation requests to an application scheduler (the application scheduler and executor are called "frames"). These allocation requests are accepted or rejected by the framework. This model is considered non-monolithic because it is a "two-level" scheduler and the scheduling algorithm is pluggable. Mesos allows any implementation of any scheduling algorithm, each algorithm can accept or reject allocation requests according to its own policy, and can accommodate thousands of schedulers running in the same cluster in a multi-tenant manner.
Mesos' two-level scheduling model allows each framework to decide (itself) which algorithm to use to schedule work to run. Mesos acts as an arbiter, scheduling resources on multiple schedulers, resolving conflicts, and ensuring that resources are distributed fairly based on business policies. When an allocation request arrives, the framework executes tasks to consume those provided resources. Or the framework may choose to reject the request and wait for the next allocation request. This model is very similar to how multiple apps run simultaneously on a laptop or smartphone. When more memory is needed, new threads or requests are created, and the OS arbitrates all requests. Years of practical development of operating systems and distributed systems have proved that the advantage of this model is that it has good scalability. It has been proven by Google and Twitter.
Scheduling of YARN
现在，我们再看一下YARN这边发生了什么。当job请求到达YARN资源管理器，YARN评估所有可用的资源然后调度job。YARN以一种整体的方式，直接决定job运行的位置。在MapReduce架构演变的过程中，重申强调YARN的出现十分重要。在Hadoop任务的资源规模伸缩需求的驱动下，YARN把资源管理的模型从MR的Job Tracker中独立出来，在Resources Manager组件中实现。
为了调度Hadoop任务，YARN进行了优化(过去一贯的Hadoop任务是持续一段时间的批处理任务)。这意味着YARN既不是为长时间运行的服务而设计，也不是为满足短期交互/快速响应式请求(像简短而快速的Spark任务)，尽管它可能调度其他种类的工作任务，但这并不是一个理想的模型。MapReduce的资源需求、执行模型和架构需求不同于长时间运行的服务，如Web服务器、SOA应用程序或是像Spark和Storm那样的实时任务。同时，YARN为了易于无状态的脚本任务重启而设计。它并不能处理像分布式文件系统或数据库那样的有状态的服务。然而YARN的整体的调度器理论上可以处理不同类型的工作负载(通过把新的算法合并到调度代码)，对于支持日益复杂的调度算法，这并不是一个轻量级的模型。
YARN vs Mesos?
在对比YARN和Mesos时，明白整体的调度能力和为什么需要两者选一十分重要。虽然有些人可能认为YARN和Mesos大同小异，但并非如此。区别在于用户一开始使用时需求模型的不同。每种模型没有明确地错误，但每种方法会产出不同的长期结果。我认为这就是选择如何使用它们的关键。Ben Hindman和Berkeley AMP实验室在设计Mesos时，与Google设计Omega的团队同期进行，Mesos系统得益于Google的Omega系统设计的经验，构建了一个更好的非单体(两阶段)的调度器。
当你把如何管理数据中心作为整体来评估时，一方面使用Mesos来管理数据中心的所有资源，另一方面使用YARN来安全的管理Hadoop任务，但它并不具有管理整个数据中心的能力。数据中心运营商倾向于把集群划分为的不同区域(Hadoop集群和非Hadoop集群)来应对这两个场景。
在同一个数据中心使用Mesos和YARN，为了受益于资源管理器，目前需要创建两个静态分区。此时意味着当指定资源被Hadoop的YARN管理时，Mesos就无法起作用。这也许过于简化了，尽管这么做确实有效。但本质上，我们是想避免这种情况。
项目Myriad介绍
这不禁让我们发问：能否让企业和数据中心受益于YARN和Mesos的协调工作?答案是肯定的。一些著名的公司——eBay、MapR和Mesosphere共同合作了一个项目叫做Myriad.
这个开源软件项目既是一个Mesos框架，又是一个YARN调度器，这就使得Mesos能够管理YARN的资源请求。当一个任务到达YARN时，它会通过Myriad调度器调度它，使请求与Mesos提供的资源匹配。相应的，Mesos也会将它传递给Mesos工作节点。之后，这个Mesos节点会把这个请求与一个正在执行YARN节点的管理器的Myriad执行器关联。Myriad在Mesos资源启动YARN节点管理器，启动之后，Mesos资源会告诉YARN资源管理器哪些资源可用。这时YARN就可以随意地使用这些资源。Myriad为Mesos的可用资源池和YARN的任务(需要用到Mesos中资源)之间架起了一座无缝连接的桥梁。
这种做法的优点是，它不仅让你在共享的集群中弹性的使用YARN，使得YARN比最初设计时更具活力和弹性。而且，它使得数据中心的运维团队在给YARN资源扩容时无需重新配置YARN集群。整个数据中心的扩容变得十分容易。该模型提供了一种简单的方式运行和管理多个YARN的实现，甚至在同一个集群上运行多个不同版本的YARN。
Myriad把YARN和Mesos两者的优势结合起来。通过使用Myriad项目，让Mesos和YARN可以协作，你可以完成一个实时业务。数据分析可以在和运行生产服务的相同硬件上执行。你不再需要面临由静态分区引起的资源限制(和低利用率)。资源可以根据业务的需求弹性的伸缩。
最后的思考
为了确保人们理解这个项目的来源，我认为Mesos和YARN擅长在自己特定的场景下工作，并且都有提升的空间。两者的资源管理器在安全领域都能有所提升;而安全的支持对企业采纳与否至关重要。
Mesos需要一个端到端的安全架构，我个人觉得可以使用Kerberos来提供安全支持，但根据个人经验，这样做应该不会简单。对Mesos其他方面的提升同样十分复杂，主要归纳为资源的抢占和撤销。假设一个业务的所有资源已经分配，当业务依赖运行的一个最重要的资源项需要扩容时，甚至这个扩容工作仅需要数十分钟来完成，你仍然会因为缺少资源而无法完成。资源的抢占和撤销就可以解决这个问题。目前，Mesos围绕着这个问题有多种解决方案，但我十分期待Mesos委员会使用Dynamic Reservations和Optimistic (Revocable) Resources Offers来解决这个问题。
Myriad作为一种新的技术，让我们把数据中心或云端的所有资源当作一个简单的资源池来使用。正如Hadoop消除数据孤岛之间的壁垒一样，Myriad消除了孤立的集群之间的壁垒。通过Myriad，开发者可以专注于业务依赖的数据和应用程序，而运维团队可以更敏捷地管理他们的计算资源。这为我们专注数据而不被基础设施持续困扰打开了另一扇窗。有了Myriad，存储网络的限制和计算与存储之间的协调就成为我们在实现完整的灵活性、敏捷和伸缩上的最后一个需要攻克的难题。
Myriad项目托管在GitHub上并提供下载。这里提供文档更详细地描述了它是如何运作的。
Original link: http://radar.oreilly.com/2015/02/a-tale-of-two-clusters-mesos-and-yarn.html
http://ju.outofmemory.cn/entry/231508
http://www.zhihu.com/question/23642987
http://blog.csdn.net/xinghun_4/article/details/47907161