Qt audio and video development 18-haikang sdk callback

I. Introduction

Haikang sdk displays real-time video streams in addition to the handle method, and also supports the callback method to get each picture and draw it by yourself. Of course, the callback method is not only to get the video data, but the audio data is also obtained together. Just call the audio device to play it. , Regarding the Haikang sdk callback, it took a while to get it done. Maybe I didn’t refer to the provided demo at the beginning and didn’t search thoroughly. I just looked at the sdk document to toss and toss, and it was a week. I didn’t get it. I found the right way later, but I almost lost it. This reminds me of many things, including things in life. Isn’t it true? When you are stubborn enough, experimenting with various methods, and you are almost exhausted to give up, in fact, you are one step away from success at this time, and you are really so missing. The same is true for handling many things in life. , So in many cases, if the direction is right, and if you persisted and worked hard, if it doesn’t work, try harder and estimate it will be ok.

After tossing for a long time to summarize where the failure is, it is correct to call NET_DVR_RealPlay_V40 to set the callback function, and the callback function is also entered. It is also correct to call PlayM4_SetDecCallBackMend to set the decoding callback function. In the end, I found that the problem lies in the decoded data, and the data is also obtained. The default is yv12 data. If you need to convert to image, you need to do a conversion. This conversion online finds a bunch of functions I came to the test, but they all failed. Later, I found a yv12 to rgb888 format, and it was finally possible, so I went.

Haikang SDK callback process:

1. Call NET_DVR_RealPlay_V40 to set the callback processing function.
2. Open, play, and decode sequentially in the callback processing function RealDataCallBack.
3. Call PlayM4_GetPort to get the unused channel number of the playback library.
4. Call PlayM4_OpenStream to open the video stream.
5. Call PlayM4_SetDecCallBackMend to set the decoding callback function, only decoding but not displaying.
6. Call PlayM4_Play to play the video stream.
7. Call PlayM4_InputData to decode data cyclically.
8. The audio and video data are processed separately in the decoding callback function DecCallBack.
9. Call the yv12ToRGB888 function packaged by yourself to convert the data into QImage.

Please note the following points about the callback function:

1. The callback function must have the keyword CALLBACK.
2. The callback function itself must be a global function or a static function, and cannot be defined as a member function of a particular class.
3. The callback function is not directly called and executed by the developer, but uses the system interface API function as the starting point.
4. The callback function is usually passed as a parameter to the system API and called by the API.
5. The callback function may be called once by the system API, or it may be called multiple times in a loop.

2. Features

1. Supports playing video streams and local MP4 files.
2. Support two modes: handle and callback.
3. Multi-threaded display images, not stuck in the main interface.
4. Reconnect the webcam automatically.
5. You can set the border size, offset and border color.
6. You can set whether to draw OSD labels, that is, label text or pictures and label positions.
7. Two OSD positions and styles can be set.
8. Can set whether to save to file and file name.
9. You can directly drag files to haikangwidget control to play.
10. Support h264/h265 video stream.
11. Can pause and resume playing.
12. Support storage of single video files and timing storage of video files.
13. Customize the top floating bar, send a click signal notification, and set whether to enable it.
14. You can set the screen stretch fill or equal proportion fill.
15. You can set the decoding to be speed priority, quality priority, and equalization processing.
16. You can take screenshots (original pictures) and screenshots (video forms) of videos.
17. The video files are stored as MP4 files.
18. Support focus control, pan/tilt control.
19. Customizable functions.

Three, renderings

Four, related sites

1. Domestic site: https://gitee.com/feiyangqingyun/QWidgetDemo
2. International site: https://github.com/feiyangqingyun/QWidgetDemo
3. Personal homepage: https://blog.csdn.net/feiyangqingyun
4. Zhihu Homepage: https://www.zhihu.com/people/feiyangqingyun/
5. Experience address: https://blog.csdn.net/feiyangqingyun/article/details/97565652

Five, the core code

//yv12转RGB888
static bool yv12ToRGB888(const unsigned char *yv12, unsigned char *rgb888, int width, int height)
{
    if ((width < 1) || (height < 1) || (yv12 == NULL) || (rgb888 == NULL)) {
        return false;
    }

    int len = width * height;
    unsigned char const *yData = yv12;
    unsigned char const *vData = &yData[len];
    unsigned char const *uData = &vData[len >> 2];

    int rgb[3];
    int yIdx, uIdx, vIdx, idx;

    for (int i = 0; i < height; ++i) {
        for (int j = 0; j < width; ++j) {
            yIdx = i * width + j;
            vIdx = (i / 2) * (width / 2) + (j / 2);
            uIdx = vIdx;

            rgb[0] = static_cast<int>(yData[yIdx] + 1.370705 * (vData[uIdx] - 128));
            rgb[1] = static_cast<int>(yData[yIdx] - 0.698001 * (uData[uIdx] - 128) - 0.703125 * (vData[vIdx] - 128));
            rgb[2] = static_cast<int>(yData[yIdx] + 1.732446 * (uData[vIdx] - 128));

            for (int k = 0; k < 3; ++k) {
                idx = (i * width + j) * 3 + k;
                if ((rgb[k] >= 0) && (rgb[k] <= 255)) {
                    rgb888[idx] = static_cast<unsigned char>(rgb[k]);
                } else {
                    rgb888[idx] = (rgb[k] < 0) ? (0) : (255);
                }
            }
        }
    }
    return true;
}

//解码回调 视频为YUV420P数据(YV12),音频为PCM数据
static void CALLBACK DecCallBack(qport nPort, char *pBuf, qport nSize, FRAME_INFO *pFrameInfo, quser luser, quser nReserved2)
{
    HaiKangThread *thread = (HaiKangThread *)luser;
    long frameType = pFrameInfo->nType;

    //视频数据是 T_YV12 音频数据是 T_AUDIO16
    if (frameType == T_YV12) {
        //qDebug() << TIMEMS << width << height << thread;
        int width = pFrameInfo->nWidth;
        int height = pFrameInfo->nHeight;
        QImage image(width, height, QImage::Format_RGB888);
        if (yv12ToRGB888((unsigned char *)pBuf, image.bits(), width, height)) {
            thread->setImage(image);
        }
    } else if (frameType == T_AUDIO16) {
        //qDebug() << TIMEMS << "T_AUDIO16" << thread;
    }
}

static void CALLBACK RealDataCallBack(LONG lRealHandle, DWORD dwDataType, BYTE *pBuffer, DWORD dwBufSize, void *dwUser)
{
    //每个类都对应自己的 port
    HaiKangThread *thread = (HaiKangThread *)dwUser;
    qport nPort = thread->port;

    DWORD dRet;
    switch (dwDataType) {
        case NET_DVR_SYSHEAD:
            //获取播放库未使用的通道号
            if (!PlayM4_GetPort(&nPort)) {
                break;
            }

            if (dwBufSize > 0) {
                thread->port = nPort;
                if (!PlayM4_OpenStream(nPort, pBuffer, dwBufSize, 1024 * 1024)) {
                    dRet = PlayM4_GetLastError(nPort);
                    break;
                }

                //设置解码回调函数 只解码不显示
                if (!PlayM4_SetDecCallBackMend(nPort, DecCallBack, (quser)dwUser)) {
                    dRet = PlayM4_GetLastError(nPort);
                    break;
                }

                //打开视频解码
                if (!PlayM4_Play(nPort, NULL)) {
                    dRet = PlayM4_GetLastError(nPort);
                    break;
                }

                //打开音频解码, 需要码流是复合流
                if (!PlayM4_PlaySound(nPort)) {
                    dRet = PlayM4_GetLastError(nPort);
                    break;
                }
            }
            break;

        case NET_DVR_STREAMDATA:
            //解码数据
            if (dwBufSize > 0 && nPort != -1) {
                BOOL inData = PlayM4_InputData(nPort, pBuffer, dwBufSize);
                while (!inData) {
                    sleep(10);
                    inData = PlayM4_InputData(nPort, pBuffer, dwBufSize);
                }
            }
            break;
    }
}