Netflix's open source culture and technology

In the previous news article "Netflix Corporate Culture and Architecture Design", we introduced Netflix's highly expanded corporate culture and architecture, and as an online movie rental provider, Netflix is ​​able to manage such a large number of users, massive data, complex complex Success under the topology structure is inseparable from the open source culture and technology behind it. Netflix uses open source or self-developed technologies to enable the company to continuously build and integrate on servers distributed in more than 50 countries around the world; it can analyze and mine the massive behavior data generated by users to better recommend and improve. Its own service; at the same time, it can protect user privacy and data through performance analysis and security auditing. Let us take an inventory of the open source technologies used or contributed by Netflix .

Big Data

Data is priceless. In order to provide customers with the ultimate service, Netflix has a complete big data technology ecosystem for user data analysis and mining. Netflix not only uses widely adopted open source technologies such as Hadoop, Hive, Pig, Parquet, Presto, and Spark, but also develops and contributes several other tools and services.

  • Genie : A set of REST-ful services customized for the Hadoop ecosystem to manage jobs and resources. It has two key services: Execution
    Service and Configuration Service. The former provides a REST-ful
    API for submitting and managing Hadoop, Hive, and Pig jobs; the latter is an efficient repository for Hadoop resources, handling metadata connections and running jobs on resources.

  • Inviso : Detailed and in-depth profiling of the performance of Hadoop jobs and clusters.

  • Lipstick : Shows the workflow of Pig jobs in a clear and visual way.

  • Aegisthus : It is a map/reduce program that can read Cassandra
    SSTable. Netflix's big data platform uses this tool every day to read data from Cassandra for analysis.

Build and deliver tools

  • Nebula : Netflix's open source collection of Gradle plugins that make it easier for developers to build, test, and deploy projects with Gradle.

  • Aminator : An EBS AMI creation tool that supports CentOS/RedHat Linux images. Images created with this tool can run on EC2 instances.

  • Asgard : Cloud deployment and management tool that works with Aminator to deploy packaged AMIs from the developer's desktop to AWS.

Common runtime services and class libraries

Most of Netflix's services are based on cloud platforms and technology stacks. The cloud platform consists of cloud services, application libraries and application containers. Each part of Netflix has corresponding technical guarantees.

  • Eureka : Netflix's cloud platform service discovery technology.

  • Archaius : Distributed configuration tool.

  • Ribbon : Resilient and intelligent process processing and service communication.

  • Hystrix : Provides reliability that a single service call does not have, and provides runtime latency isolation and fault tolerance.

  • Karyon and Governator : JVM container service.

  • Prana sidecar : Provides an in-instance proxy, supporting non-JVM runtimes.

  • Zuul : Provides services such as dynamic routing, monitoring, security, and elastic scaling around cloud deployments.

  • Fenzo : Provides more advanced scheduling and resource management functions for cloud-native frameworks, and provides plug-in implementation for packing and cluster automatic expansion. Users can implement custom scheduling optimization through custom plug-ins.

Data Persistence In the
face of trillions of behavioral data every day, no single technology can satisfy all use cases and all potential needs alone. For this reason, Netflix not only uses non-persistent memory storage solutions such as Memcached and Redis, but also uses Elastic, a searchable data store, and Cassandra and MySQL, a highly available data store. In addition, Netflix has created some other aids:

  • Raigad and Priam : Assist in deployment, management, backup and recovery of Elastic and Cassandra clusters.

  • EVCache and Dynomite : For using Memcached and Redis at scale.

  • Astyanax and Dyno client library: for better use of cloud databases.

Analytics, reliability, and performance
are critical in any company's operations. Telemetry and metrics play a vital role in any company's operations. Efficient performance commands allow engineers to quickly find what they need from a large number of metrics, making them faster and more effective. make key decisions. In addition, cost management and visualization services of resource usage in the cloud are also indispensable. The tools Netflix uses in these areas include:

  • Atlas : A time-series telemetry platform that receives over 1 billion metrics per minute

  • Edda : Service to track changes in the cloud

  • Spectator Class Library: Enables better integration of Java application code with Atlas

  • Vector : High-resolution host-level metrics at minimal cost

  • Ice : Monitors current cost and cloud usage trends, giving engineers greater control over the state of their applications in the current environment

  • Simian Army : Random failure testing of Netflix instances to verify reliability

is an increasingly important thing for companies of any type and size. Netflix has contributed a large number of security tools and solutions to the open source community, which are mainly divided into two categories. Operational tools and systems that more effectively protect large dynamic environments; one category is a security infrastructure component that provides rigorous security services for modern distributed systems.

  • Security Monkey : for detecting and securing large-scale AWS environments

  • Scumblr : Uses the Internet to conduct targeted searches to locate and investigate specific security issues

  • MSL : An extensible, flexible secure messaging protocol that addresses a broad range of secure communication use cases and needs

User Interface
Netflix members around the world watch videos on a variety of devices, including TV, mobile, and desktop. For this reason, Netflix uses advanced UI technologies such as Node.js, React, and RxJS to build rich client applications. Netflix continuously improves the product by experimenting with new ideas through data-driven A/B testing to understand the value of each feature. Additionally, Netflix uses the following tools.

  • Falcor : Allows applications to represent all remote data sources with a single JSON Graph, improving data capture efficiency

  • Restify : A Nodejs-based REST application framework that supports server-side and client-side

  • RxJS : Reactive extension library for JavaScript

via InfoQ

Guess you like