Workflow management 7k +, run over 100 million times a month, Lyft open-source platform Flyte What does it mean?

Author | Allyson Gale

Translator | Liu Chang

Edit | Jane

Produced | AI technology base camp (ID: rgznai100)

REVIEW Flyte platform can more easily create concurrent, scalable and maintainable workflow to perform machine learning and data processing. Flyte more than three years of training models and data processing experience, became pricing, positioning, ETA, autopilot and other teams practical platform. In fact, internal Lyft use Flyt E manages more than 7,000 unique workflow, running a total of more than 100,000 times per month, perform one million tasks, handling 10 million containers.

Since the data has now become the company's main assets, and therefore the implementation of large-scale computing jobs critical to the business, but from an operational perspective, but there are some problems. Extension, the supervisory computing clusters to become a burden for each product group, thus slowing down the speed of iteration, and thus slow down the pace of product innovation.

Flyte's mission is to improve development speed machine learning and data processing by abstracting these costs. Lyft team through reliable, scalable, calculate carefully designed to solve many problems, so that the team can focus on business logic. In addition,Flyte can support cross-tenant shared and reused, thus solving the problem only once. With the line between data and machine learning more and more obvious, including personnel involved in these operations, it is becoming increasingly important.

To give you a better understanding of how Flyte is the solution to all these problems, the following is an overview of some of the main features of the platform:

1、Hosted、multi-tenant、and serverless

Flyte can get rid of the trouble infrastructure, so that developers can focus on business issues. As a multi-tenant support services, you can isolate themselves repo, and without prejudice to the case of the deployment and expansion of the rest of the platform. Internet will code version control, and its dependence of the container, and every time the code is executed repeatable.

为了提供这种级别的隔离，研发团队直接将其建立在 Kubernetes 上，获得了容器化提供的所有优点：可移植性，可伸缩性，可靠性等等

2、Elastic Scale

Flyte 的主要目的就是扩展。有了完全分布式的容错控制平面，就不会出现单点故障，并且可以扩展到多个集群，数千个节点和数千个并发工作流。

Lyft 证明了该平台的扩展性，Flyte 已有三年多的训练模型和数据处理经验，成为定价，定位，ETA，自动驾驶等团队可实用的平台。实际上，Lyft 内部使用 Flyte 管理着 7000 多个独特的工作流，每月总计运行超过 100000 次，执行 100 万个任务，处理 1000 万个容器。

3、Parameters、Data Lineage、and Caching

所有 Flyte 的任务和工作流均具有强类型的输入和输出。这样就使参数化工作流程，拥有丰富的数据流，以及使用预先计算的缓存版本成为可能。例如，如果要进行超参数调优，则可以在每次运行时轻松调用不同的参数。此外，如果想调用之前已经计算过的任务，无论执行该任务的是谁，Flyte 都会巧妙地使用缓存输出，从而节省时间和金钱。

4、Versioned, Reproducible, and Shareable

Flyte 中的每个实体都是不可变的，每个更改都会明确地归为新版本。这让使用者可以轻松高效地迭代，测验和回滚工作流。此外，Flyte 支持在工作流之间共享这些版本化的任务，从而避免个人和团队之间的重复工作，加快开发周期。

5、Dynamic and extensible

Flyte 与框架无关，并且有不断增加的插件集合来满足所有工作流需求，包括 K8s 上的 Spark，AWS Batch，阵列作业，Hive Qubole，容器，Pods 等。而且也很容易贡献一个插件！用多种语言编写工作流任务也可能是有利的，因此Flyte 的 SDK 可以扩展到 Python 之外，允许进行真正的多语言编程。

附参考文章：

https://flyte.org/

https://lyft.github.io/flyte/contributor/index.html

（*本文为AI科技大本营翻译文章，转载请微信联系 1092722531）

◆

精彩推荐

◆

2020年，由 CSDN 主办的「Python开发者日」活动（Python Day）正式启动。我们将与 PyCon 官方授权的 PyCon中国社区合作，联手顶尖企业、行业与技术专家，通过精彩的技术干货内容、有趣多元化的活动等诸多体验，共同为中国 IT 技术开发者搭建专业、开放的技术交流与成长的家园。未来，我们和中国万千开发者一起分享技术、践行技术，铸就中国原创技术力量。