For more technical exchanges and job opportunities, please follow the WeChat official account of ByteDance Data Platform and reply [1] to enter the official communication group
Notebook is a development environment that supports REPL mode. The so-called "REPL" is the "read-evaluate-output" loop: input a piece of code, get the corresponding result immediately, and continue to wait for the next input. Notebooks generally make exploratory development and debugging more convenient. In the Notebook environment, users can interactively write your code, run the code, view output, visualize data and view results, making it very flexible to use.
In the field of data development, Notebooks are widely used in data cleaning and transformation, numerical simulation, statistical modeling, data visualization, building and training machine learning models, etc.
But obviously for data development, notebook alone is not enough. At present, the Volcano Engine DataLeap data R&D platform provides a series of capabilities such as task development, release scheduling, monitoring and operation, etc., and notebook is added as a task type to the DataLeap data R&D platform, so that users can have notebook interactive development. Experience and enjoy the convenience provided by the one-stop big data R&D management suite.
In the Volcano Engine DataLeap data research and development platform, the core of the development process is the task. Users can create subdirectories and tasks in the task development directory under the project, and manage their tasks through a directory tree like an IDE (Integrated Development Environment). Notebook is also a task type. Users can start an independent task Kernel environment and use Notebook like developing other ordinary tasks.
Figure: Volcano Engine DataLeap data development notebook task interface
Based on the considerations of simplifying operation and maintenance costs, reducing architectural complexity, and improving user experience, in the first half of 2021, Volcano Engine DataLeap developers made an improvement to the overall architecture. The new architecture mainly makes the following improvements, which are roughly simplified to the following figure:
-
Remove JupyterHub (https://jupyterhub.readthedocs.io/en/stable/) and change JupyterLab (https://jupyterlab.readthedocs.io/en/stable/getting_started/overview.html) to multi-instance stateless constants resident service, and implement multi-user authentication for docking with Volcano Engine DataLeap.
-
Transform the data storage that originally resided locally in JupyterLab, including user-defined configuration, session maintenance, and code file reading and writing.
-
Enterprise Gateway (EG) supports persistent Kernel, persisting Kernel remote environment metainformation in remote storage (MySQL), so that it can be reconnected when restarting, and JupyterLab can know which EG a certain Kernel needs to pass (https:/ /jupyter-enterprise-gateway.readthedocs.io/en/latest/) connection.
Figure: Overall architecture of the improved version of Notebook under Volcano Engine DataLeap
After the architecture upgrade and simplification, the stability of the entire Notebook service has been greatly improved. Due to the user-conscious upgrade, DataLeap not only improves the user experience, but also greatly reduces costs such as operation and maintenance, computing power, and manpower.
It is understood that Notebook tasks have become a more frequently used task type within ByteDance. External users can purchase Volcano Engine DataLeap, a one-stop big data R&D management suite, open an interactive analysis version, and use DataLeap's Notebook tasks.
Click to jump to
Big Data R&D Governance Suite DataLeap
to learn more
{{o.name}}
{{m.name}}