Open source background

With the rapid development of live short videos and 5G, more and more data are recorded and disseminated in the form of videos. This makes the research and application of video processing technologies, which are universal tools, SDKs and solutions for implementing various video applications, particularly important.

In the video era, multimedia development faces diverse challenges, including,

Complex scenarios : complex processing links, interdependence between algorithms and projects, and diverse business scenarios.
Extension and collaboration : For some algorithm developers, high-level languages such as Python are often used, while some multimedia processing solutions only provide C/C++ interfaces. In addition, in some cases, in order to better achieve capability reuse, it is often found that the development language used for mature capabilities is not the same as the project currently being developed. How to reuse and integrate capabilities quickly and effectively is a big problem. Engineering presents significant challenges.
Standards and compatibility : Multimedia development users often face different processing needs. Some classic and commonly used transcoding and filter capabilities are used very frequently. In order to meet industry standards, they need to invest a lot of effort in development to meet their own needs and achieve industry compliance. Standard transcoding and filter processing details; there are also some complex development situations that require interoperability between different mainstream data. Developers often need to invest a lot of energy in learning to interface with various standards and SDKs.
Heterogeneous computing : For massive video processing, calculations solely relying on multi-core CPUs sometimes cannot meet performance and cost requirements. This requires hardware acceleration in combination with co-processors such as GPUs and DSPs. How to connect heterogeneous capabilities conveniently and efficiently, including simple and easy-to-use data transmission and conversion in user processing links and high-performance asynchronous scheduling, is a challenge in the new era.

In order to solve several of the challenges mentioned above, Volcano Engine developed the BMF (Babit Multimedia Framework ) universal multimedia processing framework , which can provide simple and easy-to-use cross-language interfaces, flexible scheduling and scalability, and dynamic expansion in a modular manner. Manage and reuse the atomic capabilities of video processing, build high-performance multimedia processing links in the form of Graph/Pipeline, or implement engineering integration by directly calling individual processing capabilities, helping multimedia users to implement projects in the production environment conveniently and efficiently. In the form of open source, the BMF team hopes that by helping users solve industry pain points , it can bring more attention and feedback from users and communities, and continuously enhance the company's technical influence.

What is BMF

BMF (Babit Multimedia Framework, eight-bit multimedia processing framework) is a universal multimedia processing framework self-developed by Volcano Engine . It can provide simple and easy-to-use cross-language interfaces, flexible scheduling and scalability, and can be dynamically expanded in a modular manner. Manage and reuse the atomic capabilities of video processing, and build high-performance multimedia processing links in the form of Graph to help multimedia users quickly and efficiently implement projects into the production environment. The overall ecosystem structure of BMF is shown in the figure.

Features of the BMF framework

Simple multi-language interface : BMF provides a multi-language interface to facilitate users to integrate development in different languages according to project needs, maximizing compatibility and reuse of the entire framework.
Loosely coupled multi-language module : The module provides atomic capabilities for video processing. Users can choose Python, C++, or Go to focus on developing modules; there is loose coupling between modules and frameworks, and between modules, thereby decoupling algorithms from engineering. , decoupling processing modules developed in different languages from the development languages used in upper-layer applications, greatly enhancing expansion capabilities and diversified collaboration capabilities.
Powerful scheduling engine : BMF provides a configurable parallel scheduler, multiple usage modes and advanced features of dynamically adjusting Graph to realize a high-performance scheduling mechanism responsible for the operation of DAG/Graph/Pipeline and help users through various solutions. Smaller development costs enable functional and performance requirements in complex scenarios.
Fully compatible with FFmpeg functions and standards : BMF is fully compatible with FFmpeg's processing capabilities and indicators, such as decapsulation, decoding, filter, encoding, and encapsulation capabilities. The configuration of these processing capabilities and the ability to obtain pts, duration, bitrate, fps, etc. with FFmpeg The results with consistent indicators well meet the need to quickly integrate FFmpeg capabilities into projects.
Data conversion capabilities between multiple frameworks and heterogeneous devices : supports zero-copy conversion between multiple mainstream data formats, and is highly compatible with data connections between different modules. For example, it is necessary to use ffmpeg's encoding, decoding and filter capabilities, OpenCV's transformation processing, and more complex situations where Pytorch needs to be used for intelligent analysis; at the same time, it supports heterogeneous cross-device data transfer, pixel format conversion capabilities, and asynchronous scheduling. capabilities, thereby helping users quickly solve accelerated optimization problems in the production environment.

Application scenarios

The company has implemented video transcoding, enhancement, intelligent processing services, frame extraction and review services , live video enhanced portrait cutouts , and algorithm implementation on mobile terminals at the level of 100 million per day within the company , helping the company reduce development, computing, and bandwidth costs. , improving the collaboration and development efficiency of engineering and algorithms.

how to use

Open source address:

https://github.com/BabitMF/bmf

Follow-up plan

BMF is now open source. The project is still under active iteration. In the future, efforts will be made in heterogeneous ecological construction, more atomic capability construction and mobile terminal support to improve the multimedia development ecology and strengthen industry cooperation. Everyone is welcome to try and Participate in community building and promote technology standardization.

Thanks and communication

Everyone is welcome to conduct follow-up communication and feedback through GitHub, and you are also welcome to visit the BMF website .

Volcano engine open source general multimedia processing framework BMF