Translated from
https://medium.com/netflix-techblog/lumen-custom-self-service-dashboarding-for-netflix-8c56b541548c
Netflix generate large amounts of data. One way we obtain a useful insight is achieved by an instrument panel visual data, so that we can quickly understand large amounts of information. When operational problems occur, which is especially important because our engineers need to quickly diagnose and correct problem areas.
However, the operation is just a question Netflix dashboard potential use cases. We also use dashboards to track key business indicators and draw comparison results, monitor real-time data, and even find out whether to include burgers on the lunch menu.
In short, the dashboard is very important to Netflix , but not just any dashboard platform for us.
In particular, any dashboard platform suitable for Netflix must meet the following constraints:
- Users need to be able to build its own dynamic dashboards. Since Netflix's engineers are full-cycle developer , so we should be self-service platform, such a centralized team will not become a bottleneck to meet its service engineers have a variety of needs.
- According to the above, it must support the use of various embodiments and with the mode, so the solution has to be very flexible and allows customized dashboards created.
- We must support different transmission methods use data from many different sources, such as the transport stream by WebSockets query (after completion redirected to the cache) and a standard long-running RESTful API.
- In operation event occurs, the instrument panel need quick and responsive, so that you can quickly analyze and explore possible to identify and fix the problem.
- Must Atlas and other operations critical to Netflix technology provides first-class support .
Our platform requires highly flexible, dynamic and high performance, while providing a lot of control. Eight years ago, when we first is our operating metrics invest in a dashboard platform, there is no way to meet the above constraints on Netflix while maintaining a reasonable and cost-effective solutions. So we set up their own. It is called Lumen
Designed for the flexibility
Lumen dashboard is a platform that allows a user defined parsed browser runtime JSON profile to generate a customized dashboards.
From a higher perspective, architecture Lumen appears to be relatively simple. From back-end storage to load the configuration, and then parses it into an internal data structure. The data structure is then passed to the renderer, the renderer generates a visualization of the instrument panel and UI controls. Then, when the user is presented with one UI controls interact, the process is repeated, and the new values to the same configuration.
This simple architecture is hidden lumen configuration can be complicated. The user can configure a dashboard to which most aspects, including a display of visual effects, the use of which data source to be displayed and the UI control which can even be configured to write the variable conditions reconfigured at runtime.
Driven by JSON, our users can easily create and edit your own dashboard, but also can be generated JSON with other tools integrate well. Although these configurations can be complex, but because we have improved the lumen and eliminates the common mode required by the user, therefore, we have identified a number of core concepts can be implemented most of the other functions and modes by combining.
Thus, now, dashboard Lumen each composed mainly of the following concepts:
- Data source - define how certain types of data will be loaded into the instrument panel for use. They may be generic (e.g., REST data source) may be a type (e.g. Atlas data source). Types of data sources can be built on top of the core, common types of data sources. For example, you can Elasticsearch data source to build the REST data source, the data source is configured to publish the query to Elasticsearch API.
- Visualization - define how some data on the dashboard. From your standard line chart or bar chart to a custom form Markdown format, Lumen supports a variety of visualization. These type of visualization is independent of any specific data source, but they do require input data having a specific structure.
- Mapper - defines how to convert the payload data from the source and use as will be appreciated Lumen format, for example the desired visual effects data structure. This is how the lumen so that any data source from any manner with the use of visualization. The mapper can also be used to convert the dashboard itself or configuration data are mapped to a variable, as described below.
- Variable - defined values, which can then be partially replaced by other configurations. Variables may be statically defined in the configuration, but may be defined for a control, such as an input or switch, the user can then modify the UI. For example, you can add a variable control that allows the user to switch between viewing the production or test environment data. Variable allows you to dynamically add value to the dashboard, and even can be used to modify the mapping output in order to provide greater flexibility.
The combination of these four concepts of our users can be flexibly and dynamically build their own dashboards reactive with any desired data. For each of these concepts visualization "cell" is also rendered very well together, so that they can be reused in a plurality of instrument panel when needed.
Our users can precisely control they want and dynamic fields and values can be controlled from the user interface. This allows some users to build an instrument panel for a very specific use cases, and other users can construct an instrument panel for a very wide range of embodiments and exploratory.
While maintaining the flexibility to maintain responsiveness
Although Lumen architecture provides a flexible and dynamic platform for our users, but by default does not necessarily bring their own agility and responsive user experience. For example, the data source Lumen given little knowledge of how to run the instrument panel. They will send a large payload it? They will need a lot of client data processing it? They have reasonable delay it?
In order to meet our business needs of the user experience and provide patients with a pleasant, we soon realized that we could not get and parse the data as usual as in Web applications.
Most Web applications to access and parse the data in the main JavaScript thread of the browser. When you handle large volumes of data or complex user logic, which may lead to "chaos" in the browser, which is particularly evident when your application begins to lag or even freeze when the main thread is busy.
To avoid this problem and related issues, lumens of most data operations are web workers . When all the data is loaded dashboard, which allows the main thread can Lumen remains idle for user interaction, such as scrolling, and interact with a single graph.
展望未来
自从Lumen最初构想成为一个功能强大，灵活而又动态的平台，满足Netflix各种仪表板需求后，Lumen有了很大的发展。每周大约有1,500个唯一用户通过5,000个唯一的仪表板接收超过150,000次观看。这些视图每天从十几个不同的后端数据源中生成超过45万张图表。
该平台的灵活性极大地推动了Lumen在内部的采用，我们不断面对我们的用户，为该平台找到新的有趣的用例，包括有趣的“ hacks”，例如构建午餐食品仪表板。
随着Netflix的持续增长，我们预计对强大的仪表板平台的需求将继续增长，并为继续改进Lumen感到兴奋。我们认为流明目前有三个主要方面需要改进。
首先是减少不再适合Lumen现在所基于的核心概念的旧功能。与许多长期存在的项目一样，过去添加的某些功能不再有意义。一些解决方案只针对有限的用例，而另一些则被更通用的解决方案所取代，这些解决方案为用户提供了更多的控制权。
第二个是通过使用户更容易创建和管理仪表板来提高流明的可用性。当前，这意味着改进我们为用户提供的配置编辑工具。将来，这将意味着为用户提供一个强大的WYSIWYG风格的界面，该界面可以减轻认知负担，同时更容易发现Lumen提供的强大功能。
最后，我们要继续投资流明的可扩展性。尽管当前的体系结构可以轻松带来自己的数据源，但是创建自定义可视化效果或添加对与其他工具集成的一流支持都需要更改Lumen的源代码。我们计划使系统更加可插拔，以使用户能够扩展平台而无需维护人员的直接支持。
我们与Lumen的最终目标是应对众多用户的所有挑战，同时为他们提供体验，使他们能够专注于仪表板中的关键信息和价值，而不是所使用的工具。流明的未来是光明的！
Lumen由Netflix的Trent Willis，John Tregoning和Matthew Johnson开发和维护。 我们一直在寻找新的想法和改进方法。因此，如果您有兴趣在这个领域中贡献或只是谈论想法，请访问LinkedIn或Twitter！