Serverless practice in large-scale data processing

Source | Serverless

Author | Xiliu

Head Image | Download from Visual China

Preface

When you first contact Serverless, there is a new way of using it that is not so obvious: Compared with the traditional server-based method, the Serverless service platform can make your application scale quickly and horizontally, and the work of parallel processing is more effective. This is mainly because Serverless does not have to pay for idle resources, and there is no need to worry about insufficient reserved resources. In the traditional usage paradigm, users must reserve hundreds of servers to do some highly parallelized tasks with a short execution time, and they must pay for each server, even if some servers are no longer working. .

Take Alibaba Cloud Serverless product-function computing as an example, it can perfectly solve all your concerns above:

If your task itself is not very computationally expensive, but there are a large number of concurrent task requests that need to be processed in parallel, such as multimedia file processing, document conversion, etc.;
A task itself is very computationally intensive, requires a single task to be processed quickly, and can also support parallel processing of multiple tasks.

In this scenario, the user’s only concern is: Your task can be divided and conquered and the subtasks can be processed in parallel. A long task that takes an hour to process can be broken down into 360 independent 10 seconds. Long subtasks are processed in parallel, so that tasks that you used to take an hour to process can now be completed in only 10 seconds. As the billing model is used, the amount of calculation and the cost are roughly the same, while the traditional model will definitely be wasted because of the reserved resources, and the cost of the waste is also borne by you.

Next, I will elaborate on Serverless's practice in large-scale data processing.

Extremely flexible expansion and contraction to deal with calculation fluctuations

Before introducing relevant large-scale data processing examples, here is a brief introduction to function calculations.

1. Introduction to Function Calculation

Developers use programming languages to write applications and services. For the development languages supported by functional computing, please refer to the list of development languages;
The developer uploads the application to the function calculation;
Trigger function execution: trigger methods include OSS, API gateway, log service, table storage, function computing API, SDK, etc.;
Dynamic expansion in response to requests: Function computing can automatically expand according to the amount of user requests, and the process is transparent to you and your users;
Billing according to the actual execution time of the function: After the function is executed, the execution fee can be checked through the bill, and the charging granularity is accurate to 100 milliseconds.

Details: Function Computing Official Website

At this point, you can roughly understand how function computing works. Next, a large number of video parallel transcoding cases will be used to illustrate: Suppose a home education or entertainment-related company, teacher teaching videos or new film sources are generally centralized Generated, and you want these videos to be quickly transcoded so that customers can quickly see the video playback. For example, in the current epidemic, the number of courses generated by online education has increased sharply, and the peak of class is generally at 10 o'clock, 12 o'clock, 16 o'clock, 18 o'clock and other obvious peak periods. All new courses are processed in a specific time (such as half an hour). The uploaded video is a universal and universal requirement.

2. Flexible and highly available audio and video processing system

OSS trigger

As shown in the figure above, the user uploads a video to OSS, the OSS trigger automatically triggers the function execution, the function calculation automatically expands, the function logic in the execution environment calls FFmpeg to transcode the video, and saves the transcoded video back to OSS.

Message trigger

As shown in the above figure, the application only needs to send a message to automatically trigger the function to perform audio and video processing tasks. The function calculation is automatically expanded, and the function logic in the execution environment calls FFmpeg to transcode the video, and save the transcoded video Back to OSS.

Directly call the SDK manually to perform audio and video processing tasks

Take python as an example, roughly as follows:

 1python
 2    # -*- coding: utf-8 -*-
 3    import fc2
 4    import json
 5
 6    client = fc2.Client(endpoint="http://123456.cn-hangzhou.fc.aliyuncs.com",accessKeyID="xxxxxxxx",accessKeySecret="yyyyyy")
 7
 8    # 可选择同步/异步调用
 9    resp = client.invoke_function("FcOssFFmpeg", "transcode", payload=json.dumps(
10    {
11        "bucket_name" : "test-bucket",
12        "object_key" : "video/inputs/a.flv",
13        "output_dir" : "video/output/a_out.mp4"
14    })).data
15
16    print(resp)

We can also see from the above that there are many ways to trigger function execution. At the same time, simply configure the SLS log to quickly realize a flexible, high-availability, pay-as-you-go audio and video processing system, while providing free operation and maintenance, specific A dashboard with powerful functions such as business data visualization and powerful custom monitoring alarms.

Currently, the audio and video cases that have landed include UC, Yuque, Laiping Design House, Hupu, and several online education head customers. During peak periods, some customers flexibly use CPU computing resources of more than ten thousand cores for parallel processing. The video reaches 1700+, while providing a very high cost performance.

For details, please refer to:

simple-video-processing
fc-oss-ffmpeg

Task divide and conquer, parallel acceleration

This idea of dividing and conquering tasks is an interesting thing to apply to function calculations. Here is an example. For example, if you have a large 20G 1080P HD video that needs to be transcoded, even if you use a high-end machine, you need The time may still need to be calculated by the hour. If there is a problem and the transcoding is interrupted in the middle, you can only restart the transcoding process again. If you use the idea of divide and conquer + function calculation, the transcoding process evolves into fragmentation -> Parallel transcoding and sharding -> Merge shards , so that you can solve the two pain points mentioned above:

Sharding and composite sharding are memory-level copies, which require very little calculation, and the transcoding that really consumes calculations is split into many sub-tasks for parallel processing. In this model, the maximum time for slicing transcoding is basically Equal to the transcoding time of the entire large video;
Even if an exception occurs during the transcoding of a certain segment, you only need to retry the transcoding of this segment, without the need for the entire big task to be overwhelmed.

By reasonably decomposing large tasks, using function calculations, and writing a little code, you can quickly complete a large-scale data processing system that is flexible, highly available, parallel acceleration, and pay-as-you-go.

Before introducing this solution, let's briefly introduce the serverless workflow. The serverless workflow can organize functions and other cloud services and self-built services in an organized manner.

1. Introduction to Serverless Workflow

Serverless Workflow is a fully managed cloud service used to coordinate the execution of multiple distributed tasks. In a serverless workflow, you can arrange distributed tasks in sequence, branch, parallel, etc. The serverless workflow will reliably coordinate task execution according to the set steps, track the state transition of each task, and when necessary Execute user-defined retry logic to ensure the smooth completion of the workflow. Serverless workflow simplifies tedious tasks such as task coordination, state management, and error handling required to develop and run business processes, allowing you to focus on business logic development.

Details: Serverless workflow official website

Next, a case of fast transcoding of large videos is used to illustrate the serverless work orchestration function to realize the decomposition of large computing tasks, parallel processing of subtasks, and finally achieve the goal of quickly completing a single large task.

2. Fast multi-target format transcoding of large videos

As shown in the figure below, suppose the user uploads a video in mov format to OSS, the OSS trigger automatically triggers the function execution, the function calls FnF to execute, and FnF performs transcoding of one or more formats at the same time (by the DST_FORMATS parameter in template.yml) Control), assuming that it is configured to transcode both mp4 and flv formats.

A video file can be transcoded into various formats and various other custom processing at the same time, such as adding watermark processing or updating information to the database in the after-process;
When multiple files are uploaded to OSS at the same time, the function calculation will automatically scale and process multiple files in parallel. At the same time, the transcoding of files into multiple formats is also parallel;
Combining NAS + video slicing can solve the transcoding of super-large videos. For each video, slice processing is performed first, then transcode slices in parallel, and finally synthesize. By setting a reasonable slicing time, the transcoding speed of larger videos can be greatly accelerated ；
fnf can track the execution of each step, and can customize the retry of each step to improve the robustness of the task system, such as: retry-example

For details, please refer to: fc-fnf-video-processing

In the specific case of task division and parallel acceleration, the above shared is the decomposition of CPU-intensive tasks, but the decomposition of IO-intensive tasks can also be carried out. For example, this requirement: a 20G large file in the OSS bucket in the region of Shanghai, seconds The level is transferred back to the OSS Bucket in Hangzhou. The idea of divide and conquer can also be adopted here. After receiving the dump task, the Master function allocates the range of the large file fragmentation to each Worker sub-function, and the Worker sub-function dumps its own part of the fragment in parallel. After the function finishes running all child workers, it submits a merge fragment request to complete the entire dump task.

For details, please refer to: Use function calculation to realize multi-instance concurrency to transfer large files in seconds

to sum up

This article discusses that the Serverless service platform can make your application scale quickly and horizontally, and the work of parallel processing is more effective. It also gives specific practical cases. Whether in CPU-intensive or IO-intensive scenarios, function computing + Serverless can be a perfect solution Your following concerns:

No need to pay for idle resources
Don't worry about not enough computing resources reserved
Large computational tasks need to be processed quickly
Better task flow tracking
Complete monitoring and alarm, free operation and maintenance, business data visualization, etc.
...

The serverless audio and video processing in this article is just an example. It shows the capabilities and unique advantages of function computing and serverless workflow in offline computing scenarios. We can use divergent methods to expand the boundaries of Serverless's large-scale data processing practices, such as AI, genetic computing, scientific simulation, etc. Hope this article can attract you and start your wonderful journey of Serverless.

More reading recommendations