Summary
Usually we play engine and GUI integration needs, while using GStreamer, GStreamre will be responsible for playing and control of the media, will be responsible for handling window GUI user interaction and creating displayed. In this example we will combine GStreamer QT explains how to specify video output to the specified window, as well as information on how to use GStreamer reported to update the GUI.
Integration with GUI
We know that there are two aspects of integration with the GUI Note:
- Display window management.
Since the display window is usually created by the GUI framework, so we need to tell the specific window information GStreamer. Because each platform using a different way of passing the window handle, GStreamer provides an abstract interface (GstVideoOverlay), for shielding the platform differences, we can pass directly to the window ID GUI create GStreamer.
- Updated GUI interface
Most GUI frameworks need to do a refresh operation in the main UI thread, but the internal GStreamer may create multiple threads, which needs to be passed by GstBus and GUI comes with GStreamer communication mechanism all messages generated by the main thread to GUI , and then the GUI interface is refreshed by the main thread.
Let QT will be an example to understand how to deal with the integrated GUI GStreamer framework.
Sample Code
qtoverlay.h
#ifndef _QTOVERLAY_ #define _QTOVERLAY_ #include <gst/gst.h> #include <QWidget> #include <QPushButton> #include <QHBoxLayout> #include <QVBoxLayout> #include <QSlider> #include <QTimer> class PlayerWindow : public QWidget { Q_OBJECT public: PlayerWindow(GstElement *p); WId getVideoWId() const ; static gboolean postGstMessage(GstBus * bus, GstMessage * message, gpointer user_data); private slots: void onPlayClicked() ; void onPauseClicked() ; void onStopClicked() ; void onAlbumAvaiable(const QString &album); void onState(GstState st); void refreshSlider(); void onSeek(); void onEos(); signals: void sigAlbum(const QString &album); void sigState(GstState st); void sigEos(); private: GstElement *pipeline; QPushButton *playBt; QPushButton *pauseBt; QPushButton *stopBt; QWidget *videoWindow; QSlider *slider; QHBoxLayout *buttonLayout; QVBoxLayout *playerLayout; QTimer *timer; GstState state; gint64 totalDuration; }; #endif
qtoverlay.cpp
#include <gst/video/videooverlay.h> #include <QApplication> #include "qtoverlay.h" PlayerWindow::PlayerWindow(GstElement *p) :pipeline(p) ,state(GST_STATE_NULL) ,totalDuration(GST_CLOCK_TIME_NONE) { playBt = new QPushButton("Play"); pauseBt = new QPushButton("Pause"); stopBt = new QPushButton("Stop"); videoWindow = new QWidget(); slider = new QSlider(Qt::Horizontal); timer = new QTimer(); connect(playBt, SIGNAL(clicked()), this, SLOT(onPlayClicked())); connect(pauseBt, SIGNAL(clicked()), this, SLOT(onPauseClicked())); connect(stopBt, SIGNAL(clicked()), this, SLOT(onStopClicked())); connect(slider, SIGNAL(sliderReleased()), this, SLOT(onSeek())); buttonLayout = new QHBoxLayout; buttonLayout->addWidget(playBt); buttonLayout->addWidget(pauseBt); buttonLayout->addWidget(stopBt); buttonLayout->addWidget(slider); playerLayout = new QVBoxLayout; playerLayout->addWidget(videoWindow); playerLayout->addLayout(buttonLayout); this->setLayout(playerLayout); connect(timer, SIGNAL(timeout()), this, SLOT(refreshSlider())); connect(this, SIGNAL(sigAlbum(QString)), this, SLOT(onAlbumAvaiable(QString))); connect(this, SIGNAL(sigState(GstState)), this, SLOT(onState(GstState))); connect(this, SIGNAL(sigEos()), this, SLOT(onEos())); } WId PlayerWindow::getVideoWId() const { return videoWindow->winId(); } void PlayerWindow::onPlayClicked() { GstState st = GST_STATE_NULL; gst_element_get_state (pipeline, &st, NULL, GST_CLOCK_TIME_NONE); if (st < GST_STATE_PAUSED) { // Pipeline stopped, we need set overlay again GstElement *vsink = gst_element_factory_make ("ximagesink", "vsink"); g_object_set(GST_OBJECT(pipeline), "video-sink", vsink, NULL); WId xwinid = getVideoWId(); gst_video_overlay_set_window_handle (GST_VIDEO_OVERLAY (vsink), xwinid); } gst_element_set_state (pipeline, GST_STATE_PLAYING); } void PlayerWindow::onPauseClicked() { gst_element_set_state (pipeline, GST_STATE_PAUSED); } void PlayerWindow::onStopClicked() { gst_element_set_state (pipeline, GST_STATE_NULL); } void PlayerWindow::onAlbumAvaiable(const QString &album) { setWindowTitle(album); } void PlayerWindow::onState(GstState st) { if (state != st) { state = st; if (state == GST_STATE_PLAYING){ timer->start(1000); } if (state < GST_STATE_PAUSED){ timer->stop(); } } } void PlayerWindow::refreshSlider() { gint64 current = GST_CLOCK_TIME_NONE; if (state == GST_STATE_PLAYING) { if (!GST_CLOCK_TIME_IS_VALID(totalDuration)) { if (gst_element_query_duration (pipeline, GST_FORMAT_TIME, &totalDuration)) { slider->setRange(0, totalDuration/GST_SECOND); } } if (gst_element_query_position (pipeline, GST_FORMAT_TIME, ¤t)) { g_print("%ld / %ld\n", current/GST_SECOND, totalDuration/GST_SECOND); slider->setValue(current/GST_SECOND); } } } void PlayerWindow::onSeek() { gint64 pos = slider->sliderPosition(); g_print("seek: %ld\n", pos); gst_element_seek_simple (pipeline, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH , pos * GST_SECOND); } void PlayerWindow::onEos() { gst_element_set_state (pipeline, GST_STATE_NULL); } gboolean PlayerWindow::postGstMessage(GstBus * bus, GstMessage * message, gpointer user_data) { PlayerWindow *pw = NULL; if (user_data) { pw = reinterpret_cast<PlayerWindow*>(user_data); } switch (GST_MESSAGE_TYPE(message)) { case GST_MESSAGE_STATE_CHANGED: { GstState old_state, new_state, pending_state; gst_message_parse_state_changed (message, &old_state, &new_state, &pending_state); pw->sigState(new_state); break; } case GST_MESSAGE_TAG: { GstTagList *tags = NULL; gst_message_parse_tag(message, &tags); gchar *album= NULL; if (gst_tag_list_get_string(tags, GST_TAG_ALBUM, &album)) { pw->sigAlbum(album); g_free(album); } gst_tag_list_unref(tags); break; } case GST_MESSAGE_EOS: { pw->sigEos(); break; } default: break; } return TRUE; } int main(int argc, char *argv[]) { gst_init (&argc, &argv); QApplication app(argc, argv); app.connect(&app, SIGNAL(lastWindowClosed()), &app, SLOT(quit ())); // prepare the pipeline GstElement *pipeline = gst_parse_launch ("playbin uri=file:///home/john/video/sintel_trailer-480p.webm", NULL); // prepare the ui PlayerWindow *window = new PlayerWindow(pipeline); window->resize(900, 600); window->show(); // seg window id to gstreamer GstElement *vsink = gst_element_factory_make ("ximagesink", "vsink"); WId xwinid = window->getVideoWId(); gst_video_overlay_set_window_handle (GST_VIDEO_OVERLAY (vsink), xwinid); g_object_set(GST_OBJECT(pipeline), "video-sink", vsink, NULL); // connect to interesting signals GstBus *bus = gst_element_get_bus(pipeline); gst_bus_add_watch(bus, &PlayerWindow::postGstMessage, window); gst_object_unref(bus); // run the pipeline GstStateChangeReturn sret = gst_element_set_state (pipeline, GST_STATE_PLAYING); if (sret == GST_STATE_CHANGE_FAILURE) { gst_element_set_state (pipeline, GST_STATE_NULL); gst_object_unref (pipeline); // Exit application QTimer::singleShot(0, QApplication::activeWindow(), SLOT(quit())); } int ret = app.exec(); window->hide(); gst_element_set_state (pipeline, GST_STATE_NULL); gst_object_unref (pipeline); return ret; }
qtoverlay.pro
QT += core gui widgets TARGET = qtoverlay INCLUDEPATH += /usr/include/glib-2.0 INCLUDEPATH += /usr/lib/x86_64-linux-gnu/glib-2.0/include INCLUDEPATH += /usr/include/gstreamer-1.0 INCLUDEPATH += /usr/lib/x86_64-linux-gnu/gstreamer-1.0/include LIBS += -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -lgstvideo-1.0 SOURCES += qtoverlay.cpp HEADERS += qtoverlay.h
Were preserved above to each file, execute the following command to produce an executable program. If you can not find the file header and library files, you need to modify the contents of qtoverlay.pro file based on the actual path.
qmake -o Makefile qtoverlay.pro make
Source code analysis
// prepare the pipeline GstElement *pipeline = gst_parse_launch ("playbin uri=file:///home/jleng/video/sintel_trailer-480p.webm", NULL); // prepare the ui PlayerWindow *window = new PlayerWindow(pipeline); window->resize(900, 600); window->show();
After GStreamer is initialized in the main function and creates application objects QT is constructed Pipline, configuration GUI window object. In the constructor to initialize PlayerWindow of buttons and windows, creating regularly updated Timer progress bar.
// seg window id to gstreamer GstElement *vsink = gst_element_factory_make ("ximagesink", "vsink"); WId xwinid = window->getVideoWId(); gst_video_overlay_set_window_handle (GST_VIDEO_OVERLAY (vsink), xwinid); g_object_set(GST_OBJECT(pipeline), "video-sink", vsink, NULL); ... gst_bus_add_watch(bus, &PlayerWindow::postGstMessage, window); ... GstStateChangeReturn sret = gst_element_set_state (pipeline, GST_STATE_PLAYING); ... int ret = app.exec(); ...
Then we created a separate video window ID setting ximagesink for video rendering, and we will create Qt to GStreamer, GStreamer let get the window ID rendered, then use g_object_set () the custom Sink by "video-sink" attribute set to playbin in.
At the same time, we set the message processing function of GStreamer, all messages will be forwarded postGstMessage function. For subsequent calls GUI interface objects, we need to GUI window pointer as user-data, then converted to the GUI object postGstMessage.
Pipeline state is then set to start playback PLAYING.
Finally, the framework calls the GUI event loop, exec () will execute until the window is closed.
Because of GStreamer GstBus default main loop and use GLib event handling mechanism, it is necessary to ensure GLi default MainLoop running in a thread. In this case, Qt will automatically use the GLib main loop in Linux, so we do not need additional processing.
gboolean PlayerWindow::postGstMessage(GstBus * bus, GstMessage * message, gpointer user_data) { PlayerWindow *pw = NULL; if (user_data) { pw = reinterpret_cast<PlayerWindow*>(user_data); } switch (GST_MESSAGE_TYPE(message)) { case GST_MESSAGE_STATE_CHANGED: { GstState old_state, new_state, pending_state; gst_message_parse_state_changed (message, &old_state, &new_state, &pending_state); pw->sigState(new_state); break; } case GST_MESSAGE_TAG: { GstTagList *tags = NULL; gst_message_parse_tag(message, &tags); gchar *album= NULL; if (gst_tag_list_get_string(tags, GST_TAG_ALBUM, &album)) { pw->sigAlbum(album); g_free(album); } gst_tag_list_unref(tags); break; } case GST_MESSAGE_EOS: { pw->sigEos(); break; } default: break; } return TRUE; }
After we converted the GUI object, and then processed according to message type. In postGstMessage we did not directly update the GUI, because the GStreamer Bus deal with the GUI thread the main thread may be different threads directly update the GUI to be wrong or invalid. Thus the use of Qt signal-slot mechanism in the respective grooves on the line function updates the GUI information. Here only deal with three kinds of messages STATE_CHANGED (change of state), TAG (media metadata and encoding information), EOS (end play), GStreamer supported messages can check out the official document GstMessage .
void PlayerWindow::onPlayClicked() { GstState st = GST_STATE_NULL; gst_element_get_state (pipeline, &st, NULL, GST_CLOCK_TIME_NONE); if (st < GST_STATE_PAUSED) { // Pipeline stopped, we need set overlay again GstElement *vsink = gst_element_factory_make ("ximagesink", "vsink"); g_object_set(GST_OBJECT(pipeline), "video-sink", vsink, NULL); WId xwinid = getVideoWId(); gst_video_overlay_set_window_handle (GST_VIDEO_OVERLAY (vsink), xwinid); } gst_element_set_state (pipeline, GST_STATE_PLAYING); }
When you click on the Play button, onPlayClicked function is called, we call this state GStreamer Pipeline interface settings directly. When the show ends or click Stop, GStreamer will switch to NULL to release all the resources in the state, so we need to re-set playbin of vido-sink, and specify the video output window.
PAUSE, Stop similar process, directly call gst_element_set_state () corresponding to the Pipeline state.
void PlayerWindow::refreshSlider() { gint64 current = GST_CLOCK_TIME_NONE; if (state == GST_STATE_PLAYING) { if (!GST_CLOCK_TIME_IS_VALID(totalDuration)) { if (gst_element_query_duration (pipeline, GST_FORMAT_TIME, &totalDuration)) { slider->setRange(0, totalDuration/GST_SECOND); } } if (gst_element_query_position (pipeline, GST_FORMAT_TIME, ¤t)) { g_print("%ld / %ld\n", current/GST_SECOND, totalDuration/GST_SECOND); slider->setValue(current/GST_SECOND); } } } void PlayerWindow::onSeek() { gint64 pos = slider->sliderPosition(); g_print("seek: %ld\n", pos); gst_element_seek_simple (pipeline, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH , pos * GST_SECOND); }
We created a Timer for refresh the progress bar per second in the constructor, when refreshSlider is called, we get the total time of the file and the current time by gst_element_query_duration () and gst_element_query_position (), and refresh the progress bar. Since GStreamer return time in nanoseconds, so we need to show GST_SECOND by converting it to a second time.
We also deal with the user's Seek operation, the progress bar when you pull in to a location, get position Seek, call the gst_element_seek_simple () to jump to a specified location. We do not care about the call to GStreamer is in which thread, internal GStreamer will be processed automatically.
to sum up
In this article, we learn:
- How to use gst_video_overlay_set_window_handle () passes the GUI window handle to GStremaer.
- How to use the GUI to the message signal transmitted trough the main thread.
- How to use the Timer regularly updated GUI.
Quote
https://gstreamer.freedesktop.org/documentation/video/gstvideooverlay.html?gi-language=c
https://gstreamer.freedesktop.org/documentation/tutorials/basic/toolkit-integration.html?gi-language=c
https://doc.qt.io/qt-5/qmake-manual.html