What is Gstreamer?
Gstreamer is a support for Windows, Linux, Android, iOS cross-platform multimedia framework, the application can by way of the pipeline (Pipeline) will each step of multimedia processing in series to achieve the desired effect. Each step through the plug (plugins) means to achieve the object-based system by GObject element (Element), to facilitate the expansion of the functions.
The figure is based on a simple hierarchical Gstreamer application of the frame:
Media Applications
The top layer is applied, such as carrying tools gstreamer (gst-launch, gst-inspect the like), and based on gstreamer package library (gst-player, gst-rtsp-server, gst-editing-services, etc.) according to application of different scenarios to achieve.
Core Framework
The intermediate layer is a Core Framework, to provide:
- Upper application required interface
- Plugin Framework
- Pipline framework
- Data transmission and processing mechanisms between the various Element
- Synchronization (such as audio and video synchronization) multiple media streams (Streaming) between
- Various other tools needed library
Plugins
The lowest level for a variety of plug-ins to achieve a particular data processing and audio and video output, plug-in application does not require attention to detail, will be the Core Framework layer is responsible for loading and managing plug-ins. Main categories are:
- Protocols: responsible for the various protocol processing, file, http, rtsp and so on.
- Sources: responsible for handling the data source, alsa, v4l2, tcp / udp like.
- Formats: responsible media container processing, avi, mp4, ogg and so on.
- Codecs: codec is responsible for the media, mp3, vorbis and so on.
- Filters: responsible for media stream processing, converters, mixers, effects and so on.
- Sinks: in charge of the media stream output device or to a specified destination, alsa, xvideo, tcp / udp like.
The maturity Gstreamer frame of each module and the open source protocol used by the core and placed at different plugins source package:
- gstreamer: contains the core framework and core elements.
- gst-plugins-base: gstreamer apply the necessary plug-ins required.
- gst-plugins-good: high-quality plug-ins using the LGPL license.
- gst-plugins-ugly: high quality, but the use of plug-ins of other libraries License GPL, etc., such as using the GPL x264, x265.
- gst-plugins-bad: the quality needs to be improved plug-ins, can be moved to good mature plugins list.
- gst-libav:, to allow their use in the packaging of libav gstreamer frame.
Gstreamer basic concepts
Before further study Gstreamer, we need to know some basic concepts of gstreamer.
Element
Gstreamer Element is one of the most important object types. Implement a function of an element (a file reading, decoding, output, etc.), may need to create a plurality of element, in series and sequentially to it together to form a complete pipeline.
Pad
Pad is an element of the input / output interface, into src pad (production data) and sink pad (consumption of data) two.
Two element must be connected to it via pad, pad element has a current capability (capabilities) can handle data type, an ability by comparing src pad and sink pad that are supported when connected, to select the most appropriate data type for transmission, if the element is not supported, the program will exit. After successful connection element through the pad, the data will be transmitted from the sink to the next element on an element of the src pad then pad treated.
When the element supports a variety of data processing capability, we can specify the data type by Cap.
For example, the following command specifies the width and height Cap video, videotestsrc high will generate corresponding data according to the specified width:
gst-launch-1.0 videotestsrc ! "video/x-raw,width=1280,height=720" ! autovideosink
Bin and Pipeline
Bin is a container element for managing multiple, changing the state of the bin, the bin is automatically included to modify the state of the element, it will forward the received message. If there is no bin, we need to turn the operating element we use. By the bin reduces the complexity of the application.
Pipeline inherited from the bin, the program provides a bus for message transmission, and all sub-element for synchronization. When the pipeline is set to the state PLAYING, pipeline data will be processed by the element in more than one / new thread.
The following examples of the concepts we played through a familiar file mentioned above: the test file sintel_trailer-480p.ogv
gst-launch-1.0 filesrc location=sintel_trailer-480p.ogv ! oggdemux name=demux ! queue ! vorbisdec ! autoaudiosink demux. ! queue ! theoradec ! videoconvert ! autovideosink
When playing files above command will create the following pipeline:
可以看到这个pipeline由8个element构成，每个element都实现各自的功能：
filesrc读取文件，oggdemux解析文件，分别提取audio，video数据，queue缓存数据，vorbisdec解码audio，autoaudiosink自动选择音频设备并输出，theoradec解码video，videoconvert转换video数据格式，autovideosink自动选择显示设备并输出。
不同的element拥有不同数量及类型的pad，只有src pad的element被称为source element，只有sink pad的被称为sink element。
element可以同时拥有多个相同的pad，例如oggdemux在解析文件后，会将audio，video通过不同的pad输出。
Gstreamer数据消息交互
在pipeline运行的过程中，各个element以及应用之间不可避免的需要进行数据消息的传输，gstreamer提供了bus系统以及多种数据类型（Buffers、Events、Messages，Queries）来达到此目的：
Bus
Bus是gstreamer内部用于将消息从内部不同的streaming线程，传递到bus线程，再由bus所在线程将消息发送到应用程序。应用程序只需要向bus注册消息处理函数，即可接收到pipline中各element所发出的消息，使用bus后，应用程序就不用关心消息是从哪一个线程发出的，避免了处理多个线程同时发出消息的复杂性。
Buffers
用于从sources到sinks的媒体数据传输。
Events
用于element之间或者应用到element之间的信息传递，比如播放时的seek操作是通过event实现的。
Messages
是由element发出的消息，通过bus，以异步的方式被应用程序处理。通常用于传递errors, tags, state changes, buffering state, redirects等消息。消息处理是线程安全的。由于大部分消息是通过异步方式处理，所以会在应用程序里存在一点延迟，如果要及时的相应消息，需要在streaming线程捕获处理。
Queries
用于应用程序向gstreamer查询总时间，当前时间，文件大小等信息。
gstreamer tools
Gstreamer自带了gst-inspect-1.0和gst-launch-1.0等其他命令行工具，我们可以使用这些工具完成常见的处理任务。
gst-inspect-1.0
查看gstreamer的plugin、element的信息。直接将plugin/element的类型作为参数，会列出其详细信息。如果不跟任何参数，会列出当前系统gstreamer所能查找到的所有插件。
$ gst-inspect-1.0 playbin
gst-launch-1.0
用于创建及执行一个Pipline，因此通常使用gst-launch先验证相关功能，然后再编写相应应用。
通过上面ogg视频播放的例子，我们已经看到，一个pipeline的多个element之间通过 “!" 分隔，同时可以设置element及Cap的属性。例如：
播放音视频
gst-launch-1.0 playbin file:///home/root/test.mp4
转码
gst-launch-1.0 filesrc location=/videos/sintel_trailer-480p.ogv ! decodebin name=decode ! \
videoscale ! "video/x-raw,width=320,height=240" ! x264enc ! queue ! \
mp4mux name=mux ! filesink location=320x240.mp4 decode. ! audioconvert ! \
avenc_aac ! queue ! mux.
Streaming
#Server gst-launch-1.0 -v videotestsrc ! "video/x-raw,framerate=30/1" ! x264enc key-int-max=30 ! rtph264pay ! udpsink host=127.0.0.1 port=1234 #Client gst-launch-1.0 udpsrc port=1234 ! "application/x-rtp, payload=96" ! rtph264depay ! decodebin ! autovideosink sync=false
引用
https://gstreamer.freedesktop.org/documentation/application-development/introduction/gstreamer.html
https://gstreamer.freedesktop.org/documentation/application-development/introduction/basics.html
https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html