Qt+libvlc+rtsp: Exploration of Drawing on the Video Layer

I. Introduction

1. Origin

Recently, I was in charge of a part of the project. I needed to obtain the rtsp stream from the camera, and at the same time, it was a tcp client to receive the face recognition information, use qt to display the video and draw the face recognition frame.

2. Results

3. Environment

System environment: windows 10

Qt version: 5.8.0

vlc version: 2.2.4 (it is recommended to use libvlc version 3 or above, it is difficult to find the debug library below 3, and it is more troublesome to compile by yourself under windows)

Development tools: vs2015

4. Acknowledgments and Statements

Thanks to all the authors cited in this article, and also to my friends who helped me with their open-source spirit of sharing that helped me solve the problem.

At the same time, I would like to thank the chrysanthemum monster, who has been encouraging each other since learning programming, including this program is also solved in the middle of the night with his help.

Hereby declare: This article is not responsible for the subjective opinions and remarks of all the articles cited in this article.

2. Preparation

1. Understand the principle of video transmission

If you are interested in video transmission, you can refer to the following blog posts:

(1) Transmission principle of streaming media client

http://blog.csdn.net/jianren994/article/details/8133395

(2) RTSP/RTP media transmission and control protocol

http://blog.csdn.net/ww506772362/article/details/52609379

(3) Detailed RTSP

http://blog.csdn.net/yangzhiloveyou/article/details/10161269

2. Why use vlc

Vlc is an open source multimedia player and framework. It supports most media format playback. Although the size is not very compact, it is very powerful.

You can visit VideoLAN's official website and download this player: http://www.videolan.org/

Not just experience, it's also very helpful for your testing efforts.

3. Download and configure libvlc

This article does not provide the download address, but you can easily find the libvlc sdk, put include/ and lib/ into the project directory, and configure your project properties.

Please don't forget the plugins folder in the same directory as the sdk, it is an important basis for libvlc's ability to play, please put it in the output directory of the program.

3. Start

1. Use the library to play a rtsp streaming video

First, you need these variables:

libvlc_instance_t *_inst; // running instance of libvlc
libvlc_media_t *_media; // media ready to play
libvlc_media_player_t *_media_player; // for using vlc media player

(1) Initialize your rtsp streaming

It's worth noting that the APIs used to initialize local media are not the same as rtsp streaming.

_media = libvlc_media_new_location(_inst, addr);

(2) Add some configuration for playback

(Not measured) There may be delays for real-time rtsp streaming. Adding the following code may reduce the delay.

libvlc_media_add_option(_media, ":network-caching=300 :live-caching=300;");

(3) Initialize the player

_media_player = libvlc_media_player_new_from_media(_media);

Remember to release your media at the same time:

libvlc_media_release(_melibvlc_media_release(_media);

(4) Play media

libvlc_media_player_play(_media_player);

(5) rtsp test link

Through the above steps, if there is no problem with the rtsp link, you should be able to see the media on the desktop.

Here are several rtsp test links:

① Surveillance video of a certain road section (non-real-time, 1080P)

rtsp://202.104.126.35/demo?from=2017-02-07

②A program (real-time, 240P)

rtsp://rtsp-v3-spbtv.msk.spbtv.com/spbtv_v3_1/214_110.sdp

③A lovely animation (non-real-time, 160P)

rtsp://rtsp-v3-spbtv.msk.spbtv.com/spbtv_v3_1/214_110.sdp

2. Embed the playback window into your interface

Seeing this, if you can successfully play the media, it means that there is no problem with the basic environment configuration, so go ahead with confidence.

For most developers, the next step is to embed it in their interface!

(1) If you are just as a player

Then I suggest you to use the API:

libvlc_media_player_set_hwnd(_media_player, (void*)_wdgPlayer->winId());

This is the most convenient, fast, and stable method for general development needs.

At this point, you can think of this article as a concise tip on how libvlc connects to rtsp streaming media, and the follow-up content is just for understanding.

(2) If you need to re-develop the video

For example, in my example, a border for face recognition is added, or similar annotations. More importantly, if you want to achieve semi-transparency to prompt content on the video, then keep reading.

I have tried various methods, such as: setting a transparent background, adding QStackLayout, using QGraphicsscene to draw special-shaped windows, etc...

Similar question to this:

① Set a translucent window on the upper layer of the video played by qt+mplayer, why the window is not transparent and turns black

http://bbs.csdn.net/topics/390482549

②Multi-layer Widget is implemented in QT, and the lower-layer playing video is in a state of continuous redrawing. How to realize it?

http://www.qtcn.org/bbs/simple/?t58461.html

I believe that everyone who reads this should also be overwhelmed and can't find a better way.

The problem I encountered when developing the program: In order to add transparent content to the video, I first created a QWidget, set it to transparent, and raise(). The operating phenomenon is indeed transparent, but it is indeed "traversing" from the video.

The background color seen here is the bottommost widget color.

The possible explanation is: after using this code, the handle of the widget is handed over to the directX drawing under windows.

libvlc_media_player_set_hwnd(_media_player, (void*)_wdgPlayer->winId());

To solve this problem: you can choose not to use this api and draw it yourself; you can also generate multiple hwnd for drawing (I don’t know much about it under windows, and friends who helped me during this period have mentioned this solution, I don’t know if it is feasible, and I hope to be able to verified).

3. Give all the drawing to Qt

So, how to draw without using api? Please forgive me if I don't know the terminology very well. Here I will make it easier to understand:

For video, it consists of pictures, and we call each picture a "frame". The number of frames per second (FPS) transmitted is the criterion for evaluating the smoothness of the video.

It doesn't matter if you don't know how to encode and decode, the work that libvlc has done before has laid the foundation for us. In this case, we just need to take out the frame and draw it.

Look carefully at libvlc's api and find a way to take out frames and manipulate them:

void VideoSetCallBacks(IntPtr mediaPlayInstance, VideoLockCB lockCB, VideoUnlockCB unlockCB, VideoDisplayCB displayCB, IntPtr opaque);
IntPtr VideoLockCB(IntPtr opaque, IntPtr planes);
void VideoUnlockCB(IntPtr opaque, IntPtr picture, IntPtr planes);
void VideoDisplayCB(IntPtr opaque, IntPtr picture);

Here is a part of my modified code:

typedef struct
{
	QMutex mutex;
	CVLCPainter *painter;
	flying * pixels;
}callback_param_t;

static void* lock(void* op, void** plane)
{
	callback_param_t *p = (callback_param_t *)op;
	p->mutex.lock();
	*plane = p->pixels;

	return NULL;
}

static void unlock(void* op, void* pic, void* const* plane)
{
	callback_param_t *p = (callback_param_t *)op;
	uchar * pp = (uchar *) * plane;
	unsigned char* data = (unsigned char*)*plane;
	QImage a(data, gWidth, gHeight, QImage::Format_RGBA8888);
	p->painter->updatePic(a);
	p->mutex.unlock();
}

static void display(void* op, void* pic)
{	

}

Initialize the image buffer (other data structures are recommended instead of arrays)

void CVLCPlayer::init_vlc_param()
{
	param = new callback_param_t;
	param->painter = _vlcPainter;
	param->pixels = new uchar[gWidth * gHeight * 4];
	memset(param->pixels, 0, gWidth * gHeight * 4);
}

updatePic function

void CVLCPainter::updatePic(QImage &img)
{
	_pixmap = QPixmap::fromImage(img);
	_image = img;
	update();
}

Render in widget:

void CVLCPainter::paintEvent(QPaintEvent *event)
{
	_mutex.lock();
	QPainter painter(this);
	if (_pixmap.isNull())
	{
		_mutex.unlock();
		return;
	}
	v_width = this->rect().width();
	multi = (float)v_width / (float)gWidth;
	multi = QString::number(multi, 'f', 2).toFloat();
	v_height = (int)((float)gHeight * multi);
	o_height = (this->rect().height() - v_height) / 2;

	painter.drawImage(QRect(0, o_height, v_width, v_height), _image);
	//painter.drawPixmap(QRect(0, o_height, v_width, v_height), _pixmap);

	if (_vecRect.count() > 0 && isFaceDist)
	{
		painter.setPen(Qt::red);
		for (int i = 0; i < _vecRect.count(); ++i)
			painter.drawRect(*_vecRect.at(i));
	}
	_mutex.unlock();
}

The above principles can be viewed in the blog post: http://www.cnblogs.com/smartsensor/p/4343769.html

Provide a post here, and thank the author for the source code: http://bbs.csdn.net/topics/390817375

3. Emergence, solution and re-exploration of new problems

Drawing with drawPixmap() may cause the program to crash?

There was a crash problem on my machine. Since the release version can only track drawPixmap(), the problem no longer occurs after replacing it with drawImage(). Through the difference between the two under windows, it may be due to the fact that QImage is independent of hardware, and it is also a QPaintDevice, which does not need to process images in the UI thread. But this statement is not completely based, if you are interested, you can also inform or infer possible reasons.

Bloggers have limited ability and limited experience. This article may have many problems and is only a reference direction for solving problems. Please correct me if I find any mistakes, thank you.