DeepStream初步学习

业界资讯 2020-02-07 10:02:49 阅读次数: 0

一、简介

什么是DeepStream应用程序？

DeepStream应用程序将深度神经网络和其他复杂的处理任务引入到流处理管道中，以实现对视频和其他传感器数据的近实时分析。从这些传感器中提取有意义的见解为提高运营效率和安全性创造了机会。例如，摄像头是当前使用最多的物联网传感器。在我们的家中，街道上，停车场，大型购物中心，仓库，工厂中都可以找到相机–无处不在。视频分析的潜在用途是巨大的：访问控制，防止丢失，自动结帐，监视，安全，自动检查（QA），包裹分类（智能物流），交通控制/工程，工业自动化等。
在这里插入图片描述
更具体地说，DeepStream应用程序是一组模块化插件，这些插件相连接以形成处理管道。每个插件代表一个功能块，例如，使用TensorRT进行推理或多流解码。硬件加速的插件与基础硬件（如果适用）交互以提供最佳性能。例如，解码插件与NVDEC交互，推理插件与GPU或DLA交互。每个插件可以根据需要在管道中实例化多次。
在这里插入图片描述在这里插入图片描述

什么是DeepStream SDK？

NVIDIA DeepStream SDK是基于开源GStreamer多媒体框架的流分析工具包。DeepStream SDK加快了可伸缩IVA应用程序的开发速度，使开发人员更容易构建核心深度学习网络，而不必从头开始设计端到端应用程序。包含NVIDIA Jetson模块或NVIDIA dGPU适配器的系统均支持该SDK。它由可扩展的硬件加速插件集合组成，这些插件与低级库进行交互以优化性能，并定义了标准化的元数据结构，可实现自定义/用户特定的添加。

有关DeepStream SDK的更多详细信息和说明，请参考以下材料：
NVIDIA DeepStream SDK开发指南
NVIDIA DeepStream插件手册
NVIDIA DeepStream SDK API参考文档

DeepStream SDK参考应用程序

DeepStream SDK随附了多个测试应用程序，包括预训练的模型，示例配置文件和可用于运行这些应用程序的示例视频流。其他示例和源代码示例为大多数IVA用例提供了足够的信息，以加快开发工作。测试应用程序演示：

如何使用DeepStream元素（例如，获取源代码，对多个流进行解码和多路复用，在经过预训练的模型上进行推理，对图像进行注释和渲染）
如何生成一批帧并对其进行推断以提高资源利用率
如何将自定义/特定于用户的元数据添加到DeepStream的任何组件中
以及更多… 有关完整的详细信息，请参阅《 NVIDIA DeepStream SDK开发指南》。

GStreamer插件:

GStreamer是用于插件，数据流和媒体类型处理/协商的框架。它用于创建流媒体应用程序。插件是在运行时动态加载的共享库，可以独立扩展和升级。当安排并链接在一起时，插件形成处理流水线，该流水线定义了流媒体应用程序的数据流。您可以通过其广泛的在线文档，从“什么是GStreamer？”开始了解有关GStreamer的更多信息。
除了在GStreamer框架库中提供的开源插件之外，DeepStream SDK还包括利用GPU功能的NVIDIA硬件加速插件。有关DeepStream GStreamer插件的完整列表，请参见《 NVIDIA DeepStream插件手册》。

开源GStreamer插件：

  • GstFileSrc-从文件中读取数据：视频数据或图像。
  • GstH264Parse-解析传入的H264流。对于H265编解码器，请使用H265Parse。
  • GstRtpH264Pay-将H264编码的有效负载转换为RTP数据包（RFC 3984）。
  • GstUDPSink-将UDP数据包发送到网络。与RTP有效负载（GstRtpH264Pay）配对时，它可以实现RTP流。
  • GstCapsFilter-在不修改数据的情况下限制数据格式。
  • GstV4l2Src-从v4l2设备捕获视频。
  • GstQTMux-将流（音频和视频）合并到QuickTime（.mov）文件中。
  • GstFileSink-将传入数据写入本地文件系统中的文件。
  • GstURIDecodeBin-将数据从URI解码到原始媒体中。它选择可以处理给定“ uri”方案的源元素，并将其连接到解码器。

NVIDIA硬件加速插件：

  • Gst-nvstreammux-在发送AI推理之前批处理流。
  • Gst-nvinfer-使用TensorRT运行推理。
  • Gst-nvvideo4linux2-使用硬件加速解码器（NVDEC）解码视频流；使用硬件加速编码器（NVENC）将I420格式的RAW数据编码为H264或H265输出视频流。
  • Gst-nvvideoconvert-执行视频颜色格式转换。Gst-nvdsosd插件之前的第一个Gst-nvvideoconvert插件将流数据从I420转换为RGBA，Gst-nvdsosd插件将Gst-nvdsosd插件将数据从RGBA转换为I420。
  • Gst-nvdsosd-绘制边界框，文本和关注区域（ROI）多边形。
  • Gst-nvtracker-跟踪帧之间的对象。
  • Gst-nvmultistreamtiler-从批处理缓冲区合成2D切片。
  • Gst-nvv4l2decoder-解码视频流。
  • Gst-Nvv4l2h264enc-编码视频流。
  • Gst-NvArgusCameraSrc-提供使用Argus API控制ISP属性的选项。

视频分析

使用DeepStream SDK将视频转换为分析数据

DeepStream SDK通用流分析架构定义了可扩展的视频处理管道，可用于执行推理，对象跟踪和报告。当DeepStream应用程序分析每个视频帧时，插件会提取信息并将其存储为级联元数据记录的一部分，从而保持记录与源帧的关联。管道末尾的完整元数据集合表示深度学习模型和其他分析插件从框架中提取的完整信息集。DeepStream应用程序可以使用此信息进行显示，也可以作为消息的一部分在外部进行传输，以进行进一步的分析或长期归档。
在这里插入图片描述
DeepStream对元数据使用可扩展的标准结构。基本的元数据结构NvDsBatchMeta始于在所需Gst-nvstreammux插件内部创建的批处理级元数据。辅助元数据结构保存框架，对象，分类器和标签数据。DeepStream还提供了一种在批处理，框架或对象级别添加用户特定的元数据的机制。
在这里插入图片描述
有关元数据结构和使用的更多信息，请参阅《 DeepStream插件手册》。

ps：有需要的话可能会用到JupyterLab：
Jetson 安装 Jupyter lab
TX2入门教程软件篇-安装Jupyter

二、下载

官网：https://developer.nvidia.com/deepstream-download
下载v4.0版本（需要登录）：https://developer.nvidia.com/deepstream-402-jetson-deb
我实际使用的deepstream是在Jetson Xavier上的deepstream，是刷JetPack-4.3系统时一起刷入的，同样也是v4.0.2版本。

三、官方demo

1. 直接运行官方demo
# 我是在Xavier上研究的，deepstream路径在/opt/nvidia下
$ cd /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app
# 首次运行仅仅会生成模型文件，需要二次运行
$ deepstream-app -c source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt
$ deepstream-app -c source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt
# 当然，我们也可以用：source12_1080p_dec_infer-resnet_tracker_tiled_display_fp16_tx2.txt

在这里插入图片描述

# 我们还可以查看该配置文件：
$ vim source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt

# 可以看到使用的视频源
[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../streams/sample_1080p_h264.mp4
num-sources=8
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

# 根据了解到的消息，如果运行之后没有任何日志输出，可以修改sink0里的type改成2、sync改成0（这些参数是开发人员一个一个修改出来的）
[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=5
sync=1
source-id=0
gpu-id=0
qos=0
nvbuf-memory-type=0
overlay-id=1

# 当然你也可以打开sink1（enable=1），可以保存为out.mp4文件。
[sink1]
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
sync=0
#iframeinterval=10
bitrate=2000000
output-file=out.mp4
source-id=0

配置文件内容明细请查阅：https://docs.nvidia.com/metropolis/deepstream/4.0/dev-guide/index.html#page/DeepStream_Development_Guide%2Fdeepstream_app_config.3.2.html%23wwpID0E0WB0HA

示例/配置：参考应用程序的配置文件：

  • source30_1080p_resnet_dec_infer_tiled_display_int8.txt：演示具有主要推理功能的30个流解码。（仅适用于dGPU和Jetson AGX Xavier平台。）
  • source4_1080p_resnet_dec_infer_tiled_display_int8.txt：演示具有主要推理，对象跟踪和三个不同辅助分类器的四个流解码。（仅适用于dGPU和Jetson AGX Xavier平台。）
  • source4_1080p_resnet_dec_infer_tracker_sgie_tiled_display_int8_gpu1.txt：在GPU 1上针对主要推理，对象跟踪和三个不同的二级分类器演示四个流解码（对于具有多个GPU卡的系统）。仅适用于dGPU平台。
  • config_infer_primary.txt：将 nvinfer元素配置为主要检测器。
  • config_infer_secondary_carcolor.txt， config_infer_secondary_carmake.txt， config_infer_secondary_vehicletypes.txt：将 nvinfer元素配置为辅助分类器。
  • iou_config.txt：配置一个低级的IOU（联合路口）跟踪器。
  • source1_usb_dec_infer_resnet_int8.txt：演示一台USB摄像机作为输入。
  • source1_csi_dec_infer_resnet_int8.txt：演示一个CSI摄像机作为输入；仅限于Jetson。
  • source2_csi_usb_dec_infer_resnet_int8.txt：演示一台CSI摄像机和一台USB摄像机作为输入；仅限于Jetson。
  • source6_csi_dec_infer_resnet_int8.txt：演示六个CSI摄像机作为输入；仅限于Jetson。
  • source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt：演示8解码+推断+跟踪器；仅适用于Jetson Nano。
  • source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_tx1.txt：演示8解码+推断+跟踪器；仅适用于Jetson TX1。
  • source12_1080p_dec_infer-resnet_tracker_tiled_display_fp16_tx2.txt：演示12个解码+推断+跟踪器；仅适用于Jetson TX2。
2. 修改配置文件来调用rtsp摄像头

使用下边这个配置文件，可以调用自己的网络摄像头：
其中做过的修改：

  • [tiled-display]下修改了输出显示：
    rows=1
    columns=1
  • [source0]下修改了type类型、视频源、个数：
    type=4
    uri=rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    num-sources=1
  • [sink0]修改了type=2（经测试应该是视频输出变为了窗口型，nomachine也能看到效果了）
    type=2
  • [sink1]修改了enable=1和输出文件名
    enable=1
    output-file=out_toson.mp4
# Copyright (c) 2019 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=0
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0
#qos=0
nvbuf-memory-type=0
#overlay-id=1

[sink1]
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
sync=0
#iframeinterval=10
bitrate=2000000
output-file=out_toson.mp4
source-id=0

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
sync=0
bitrate=4000000
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=8
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_fp16.engine
batch-size=8
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=4
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_nano.txt

[tracker]
enable=1
tracker-width=480
tracker-height=272
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
#ll-config-file required for IOU only
#ll-config-file=iou_config.txt
gpu-id=0

[tests]
file-loop=0

三、例程源码

1. 在源文件目录中找到源码，我们可以自己编译并运行demo
# 源文件目录（我们可以自己编译并运行demo）
$ cd /opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/deepstream-test1
$ ls
deepstream_test1_app.c  dstest1_pgie_config.txt  Makefile  README
$ make
$ ls
deepstream-test1-app    deepstream_test1_app.o   Makefile
deepstream_test1_app.c  dstest1_pgie_config.txt  README
# 运行deepstream-test1-app
$ ./deepstream-test1-app /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.h264

# demo里的视频图像文件
$ cd /opt/nvidia/deepstream/deepstream-4.0/samples/streams
$ ls
sample_1080p_h264.mp4  sample_720p.h264  sample_720p.mjpeg  sample_cam6.mp4
sample_1080p_h265.mp4  sample_720p.jpg   sample_720p.mp4    sample_industrial.jpg
Stream Type of Stream
sample_1080p_h264.mp4 H264 containerized stream
sample_1080p_h265.mp4 H265 containerized stream
sample_720p.h264 H264 elementary stream
sample_720p.jpg JPEG image
sample_720p.mjpeg MJPEG stream
sample_cam6.mp4 H264 containerized stream (360D camera stream)
sample_industrial.jpg JPEG image

在这里插入图片描述
日志信息：

Now playing: /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.h264

Using winsys: x11 
Opening in BLOCKING MODE 
Creating LL OSD context new
0:00:01.658822508 30700   0x558a9bcd20 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:01:03.125062075 30700   0x558a9bcd20 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_int8.engine
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Creating LL OSD context new
Frame Number = 0 Number of objects = 5 Vehicle Count = 3 Person Count = 2
Frame Number = 1 Number of objects = 5 Vehicle Count = 3 Person Count = 2
Frame Number = 2 Number of objects = 5 Vehicle Count = 3 Person Count = 2
..
Frame Number = 3 Number of objects = 7 Vehicle Count = 4 Person Count = 3
Frame Number = 4 Number of objects = 6 Vehicle Count = 4 Person Count = 2
Frame Number = 1441 Number of objects = 0 Vehicle Count = 0 Person Count = 0
End of stream
Returned, stopping playback
Deleting pipeline
2. 源文件

（其中Makefile和dstest1_pgie_config.txt略有修改，include文件我是拷贝出来的，模型使用绝对地址）

deepstream_test1_app.c

/*
 * Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
 * to deal in the Software without restriction, including without limitation
 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
 * and/or sell copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
 * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 * DEALINGS IN THE SOFTWARE.
 */

#include <gst/gst.h>
#include <glib.h>
#include <stdio.h>
#include "gstnvdsmeta.h"

#define MAX_DISPLAY_LEN 64

#define PGIE_CLASS_ID_VEHICLE 0
#define PGIE_CLASS_ID_PERSON 2

/* The muxer output resolution must be set if the input streams will be of
 * different resolution. The muxer will scale all the input frames to this
 * resolution. */
#define MUXER_OUTPUT_WIDTH 1920
#define MUXER_OUTPUT_HEIGHT 1080

/* Muxer batch formation timeout, for e.g. 40 millisec. Should ideally be set
 * based on the fastest source's framerate. */
#define MUXER_BATCH_TIMEOUT_USEC 4000000

gint frame_number = 0;
gchar pgie_classes_str[4][32] = { "Vehicle", "TwoWheeler", "Person",
  "Roadsign"
};

/* osd_sink_pad_buffer_probe  will extract metadata received on OSD sink pad
 * and update params for drawing rectangle, object information etc. */

static GstPadProbeReturn
osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
    GstBuffer *buf = (GstBuffer *) info->data;
    guint num_rects = 0; 
    NvDsObjectMeta *obj_meta = NULL;
    guint vehicle_count = 0;
    guint person_count = 0;
    NvDsMetaList * l_frame = NULL;
    NvDsMetaList * l_obj = NULL;
    NvDsDisplayMeta *display_meta = NULL;

    NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);

    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
        int offset = 0;
        for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
                l_obj = l_obj->next) {
            obj_meta = (NvDsObjectMeta *) (l_obj->data);
            if (obj_meta->class_id == PGIE_CLASS_ID_VEHICLE) {
                vehicle_count++;
                num_rects++;
            }
            if (obj_meta->class_id == PGIE_CLASS_ID_PERSON) {
                person_count++;
                num_rects++;
            }
        }
        display_meta = nvds_acquire_display_meta_from_pool(batch_meta);
        NvOSD_TextParams *txt_params  = &display_meta->text_params[0];
        display_meta->num_labels = 1;
        txt_params->display_text = g_malloc0 (MAX_DISPLAY_LEN);
        offset = snprintf(txt_params->display_text, MAX_DISPLAY_LEN, "Person = %d ", person_count);
        offset = snprintf(txt_params->display_text + offset , MAX_DISPLAY_LEN, "Vehicle = %d ", vehicle_count);

        /* Now set the offsets where the string should appear */
        txt_params->x_offset = 10;
        txt_params->y_offset = 12;

        /* Font , font-color and font-size */
        txt_params->font_params.font_name = "Serif";
        txt_params->font_params.font_size = 10;
        txt_params->font_params.font_color.red = 1.0;
        txt_params->font_params.font_color.green = 1.0;
        txt_params->font_params.font_color.blue = 1.0;
        txt_params->font_params.font_color.alpha = 1.0;

        /* Text background color */
        txt_params->set_bg_clr = 1;
        txt_params->text_bg_clr.red = 0.0;
        txt_params->text_bg_clr.green = 0.0;
        txt_params->text_bg_clr.blue = 0.0;
        txt_params->text_bg_clr.alpha = 1.0;

        nvds_add_display_meta_to_frame(frame_meta, display_meta);
    }

    g_print ("Frame Number = %d Number of objects = %d "
            "Vehicle Count = %d Person Count = %d\n",
            frame_number, num_rects, vehicle_count, person_count);
    frame_number++;
    return GST_PAD_PROBE_OK;
}

static gboolean
bus_call (GstBus * bus, GstMessage * msg, gpointer data)
{
  GMainLoop *loop = (GMainLoop *) data;
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_EOS:
      g_print ("End of stream\n");
      g_main_loop_quit (loop);
      break;
    case GST_MESSAGE_ERROR:{
      gchar *debug;
      GError *error;
      gst_message_parse_error (msg, &error, &debug);
      g_printerr ("ERROR from element %s: %s\n",
          GST_OBJECT_NAME (msg->src), error->message);
      if (debug)
        g_printerr ("Error details: %s\n", debug);
      g_free (debug);
      g_error_free (error);
      g_main_loop_quit (loop);
      break;
    }
    default:
      break;
  }
  return TRUE;
}

int
main (int argc, char *argv[])
{
  GMainLoop *loop = NULL;
  GstElement *pipeline = NULL, *source = NULL, *h264parser = NULL,
      *decoder = NULL, *streammux = NULL, *sink = NULL, *pgie = NULL, *nvvidconv = NULL,
      *nvosd = NULL;
#ifdef PLATFORM_TEGRA
  GstElement *transform = NULL;
#endif
  GstBus *bus = NULL;
  guint bus_watch_id;
  GstPad *osd_sink_pad = NULL;

  /* Check input arguments */
  if (argc != 2) {
    g_printerr ("Usage: %s <H264 filename>\n", argv[0]);
    return -1;
  }

  /* Standard GStreamer initialization */
  gst_init (&argc, &argv);
  loop = g_main_loop_new (NULL, FALSE);

  /* Create gstreamer elements */
  /* Create Pipeline element that will form a connection of other elements */
  pipeline = gst_pipeline_new ("dstest1-pipeline");

  /* Source element for reading from the file */
  source = gst_element_factory_make ("filesrc", "file-source");

  /* Since the data format in the input file is elementary h264 stream,
   * we need a h264parser */
  h264parser = gst_element_factory_make ("h264parse", "h264-parser");

  /* Use nvdec_h264 for hardware accelerated decode on GPU */
  decoder = gst_element_factory_make ("nvv4l2decoder", "nvv4l2-decoder");

  /* Create nvstreammux instance to form batches from one or more sources. */
  streammux = gst_element_factory_make ("nvstreammux", "stream-muxer");

  if (!pipeline || !streammux) {
    g_printerr ("One element could not be created. Exiting.\n");
    return -1;
  }

  /* Use nvinfer to run inferencing on decoder's output,
   * behaviour of inferencing is set through config file */
  pgie = gst_element_factory_make ("nvinfer", "primary-nvinference-engine");

  /* Use convertor to convert from NV12 to RGBA as required by nvosd */
  nvvidconv = gst_element_factory_make ("nvvideoconvert", "nvvideo-converter");

  /* Create OSD to draw on the converted RGBA buffer */
  nvosd = gst_element_factory_make ("nvdsosd", "nv-onscreendisplay");

  /* Finally render the osd output */
#ifdef PLATFORM_TEGRA
  transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
#endif
  sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");

  if (!source || !h264parser || !decoder || !pgie
      || !nvvidconv || !nvosd || !sink) {
    g_printerr ("One element could not be created. Exiting.\n");
    return -1;
  }

#ifdef PLATFORM_TEGRA
  if(!transform) {
    g_printerr ("One tegra element could not be created. Exiting.\n");
    return -1;
  }
#endif

  /* we set the input filename to the source element */
  g_object_set (G_OBJECT (source), "location", argv[1], NULL);

  g_object_set (G_OBJECT (streammux), "width", MUXER_OUTPUT_WIDTH, "height",
      MUXER_OUTPUT_HEIGHT, "batch-size", 1,
      "batched-push-timeout", MUXER_BATCH_TIMEOUT_USEC, NULL);

  /* Set all the necessary properties of the nvinfer element,
   * the necessary ones are : */
  g_object_set (G_OBJECT (pgie),
      "config-file-path", "dstest1_pgie_config.txt", NULL);

  /* we add a message handler */
  bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
  bus_watch_id = gst_bus_add_watch (bus, bus_call, loop);
  gst_object_unref (bus);

  /* Set up the pipeline */
  /* we add all elements into the pipeline */
#ifdef PLATFORM_TEGRA
  gst_bin_add_many (GST_BIN (pipeline),
      source, h264parser, decoder, streammux, pgie,
      nvvidconv, nvosd, transform, sink, NULL);
#else
  gst_bin_add_many (GST_BIN (pipeline),
      source, h264parser, decoder, streammux, pgie,
      nvvidconv, nvosd, sink, NULL);
#endif

  GstPad *sinkpad, *srcpad;
  gchar pad_name_sink[16] = "sink_0";
  gchar pad_name_src[16] = "src";

  sinkpad = gst_element_get_request_pad (streammux, pad_name_sink);
  if (!sinkpad) {
    g_printerr ("Streammux request sink pad failed. Exiting.\n");
    return -1;
  }

  srcpad = gst_element_get_static_pad (decoder, pad_name_src);
  if (!srcpad) {
    g_printerr ("Decoder request src pad failed. Exiting.\n");
    return -1;
  }

  if (gst_pad_link (srcpad, sinkpad) != GST_PAD_LINK_OK) {
      g_printerr ("Failed to link decoder to stream muxer. Exiting.\n");
      return -1;
  }

  gst_object_unref (sinkpad);
  gst_object_unref (srcpad);

  /* we link the elements together */
  /* file-source -> h264-parser -> nvh264-decoder ->
   * nvinfer -> nvvidconv -> nvosd -> video-renderer */

  if (!gst_element_link_many (source, h264parser, decoder, NULL)) {
    g_printerr ("Elements could not be linked: 1. Exiting.\n");
    return -1;
  }

#ifdef PLATFORM_TEGRA
  if (!gst_element_link_many (streammux, pgie,
      nvvidconv, nvosd, transform, sink, NULL)) {
    g_printerr ("Elements could not be linked: 2. Exiting.\n");
    return -1;
  }
#else
  if (!gst_element_link_many (streammux, pgie,
      nvvidconv, nvosd, sink, NULL)) {
    g_printerr ("Elements could not be linked: 2. Exiting.\n");
    return -1;
  }
#endif

  /* Lets add probe to get informed of the meta data generated, we add probe to
   * the sink pad of the osd element, since by that time, the buffer would have
   * had got all the metadata. */
  osd_sink_pad = gst_element_get_static_pad (nvosd, "sink");
  if (!osd_sink_pad)
    g_print ("Unable to get sink pad\n");
  else
    gst_pad_add_probe (osd_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
        osd_sink_pad_buffer_probe, NULL, NULL);

  /* Set the pipeline to "playing" state */
  g_print ("Now playing: %s\n", argv[1]);
  gst_element_set_state (pipeline, GST_STATE_PLAYING);

  /* Wait till pipeline encounters an error or EOS */
  g_print ("Running...\n");
  g_main_loop_run (loop);

  /* Out of the main loop, clean up nicely */
  g_print ("Returned, stopping playback\n");
  gst_element_set_state (pipeline, GST_STATE_NULL);
  g_print ("Deleting pipeline\n");
  gst_object_unref (GST_OBJECT (pipeline));
  g_source_remove (bus_watch_id);
  g_main_loop_unref (loop);
  return 0;
}

Makefile

################################################################################
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

APP:= deepstream-test1-app

TARGET_DEVICE = $(shell gcc -dumpmachine | cut -f1 -d -)

NVDS_VERSION:=4.0

LIB_INSTALL_DIR?=/opt/nvidia/deepstream/deepstream-$(NVDS_VERSION)/lib/

ifeq ($(TARGET_DEVICE),aarch64)
  CFLAGS:= -DPLATFORM_TEGRA
endif

SRCS:= $(wildcard *.c)

INCS:= $(wildcard *.h)

PKGS:= gstreamer-1.0

OBJS:= $(SRCS:.c=.o)

CFLAGS+= -I./includes

CFLAGS+= `pkg-config --cflags $(PKGS)`

LIBS:= `pkg-config --libs $(PKGS)`

LIBS+= -L$(LIB_INSTALL_DIR) -lnvdsgst_meta -lnvds_meta \
       -Wl,-rpath,$(LIB_INSTALL_DIR)

all: $(APP)

%.o: %.c $(INCS) Makefile
	$(CC) -c -o $@ $(CFLAGS) $<

$(APP): $(OBJS) Makefile
	$(CC) -o $(APP) $(OBJS) $(LIBS)

clean:
	rm -rf $(OBJS) $(APP)

dstest1_pgie_config.txt

################################################################################
# Copyright (c) 2018-2019, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

# Following properties are mandatory when engine files are not specified:
#   int8-calib-file(Only in INT8)
#   Caffemodel mandatory properties: model-file, proto-file, output-blob-names
#   UFF: uff-file, input-dims, uff-input-blob-name, output-blob-names
#   ONNX: onnx-file
#
# Mandatory properties for detectors:
#   num-detected-classes
#
# Optional properties for detectors:
#   enable-dbscan(Default=false), interval(Primary mode only, Default=0)
#   custom-lib-path,
#   parse-bbox-func-name
#
# Mandatory properties for classifiers:
#   classifier-threshold, is-classifier
#
# Optional properties for classifiers:
#   classifier-async-mode(Secondary mode only, Default=false)
#
# Optional properties in secondary mode:
#   operate-on-gie-id(Default=0), operate-on-class-ids(Defaults to all classes),
#   input-object-min-width, input-object-min-height, input-object-max-width,
#   input-object-max-height
#
# Following properties are always recommended:
#   batch-size(Default=1)
#
# Other optional properties:
#   net-scale-factor(Default=1), network-mode(Default=0 i.e FP32),
#   model-color-format(Default=0 i.e. RGB) model-engine-file, labelfile-path,
#   mean-file, gie-unique-id(Default=0), offsets, gie-mode (Default=1 i.e. primary),
#   custom-lib-path, network-mode(Default=0 i.e FP32)
#
# The values in the config file are overridden by values set through GObject
# properties.

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
#model-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_int8.engine
model-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.prototxt
labelfile-path=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/cal_trt.bin
batch-size=1
network-mode=1
num-detected-classes=4
interval=0
gie-unique-id=1
output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid

[class-attrs-all]
threshold=0.2
eps=0.2
group-threshold=1

四、其他的例程源码简介

参考官方文档“Reference Application Source Details”部分：https://docs.nvidia.com/metropolis/deepstream/dev-guide/
也可以参考博客《关于NVIDIA Deepstream SDK压箱底的资料都在这里了》：https://cloud.tencent.com/developer/article/1524712

Sample APP包括：
（注：其中都需要make编译，具体请参阅其中的README文档）

  • DeepStream Sample App
    <DS installation dir>/sources/apps/sample_apps/deepstream-app
    说明:端到端示例演示了4级联神经网络(1个一级检测器和3个二级分类器)的多相机流，并显示平铺输出。
  • DeepStream Test 1
    <DS installation dir>/sources/apps/sample_apps/deepstream-test1
    说明：对单一H264视频流，应用filesrc, decode, nvstreammmux, nvinfer(主检测网络), nvosd, renderer等DeepStream插件(元素）
  • DeepStream Test 2
    <DS installation dir>/sources/apps/sample_apps/deepstream-test2
    说明:简单的应用程序，建立在test1之上，显示额外的属性，如跟踪和二级分类属性。
  • DeepStream Test 3
    <DS installation dir>/sources/apps/sample_apps/deepstream-test3
    说明:简单的应用程序，建立在test1的基础上，显示多个输入源和批处理使用nvstreammuxer。
  • DeepStream Test 4
    <DS installation dir>/sources/apps/sample_apps/deepstream-test4
    说明：这是在Test1示例的基础上构建的，演示了“nvmsgconv”和“nvmsgbroker”插件在物联网连接管道中的使用。对于test4，用户必须修改kafka代理连接字符串才能成功连接。需要安装分析服务器docker之前运行test4。DeepStream分析文档有关于设置分析服务器的更多信息。
  • FasterRCNN Object Detector
    <DS installation dir>/sources/objectDetector_FasterRCNN
    说明:FasterRCNN物体探测器实例。
  • SSD Object Detector
    <DS installation dir>/sources/objectDetector_SSD
    说明：SSD目标探测器实例。
Tosonw
发布了25 篇原创文章 · 获赞 12 · 访问量 2万+
私信 关注

猜你喜欢

转载自blog.csdn.net/Tosonw/article/details/104154090
今日推荐
开源日报 | Chrome内置Gemini的意义不在于Gemini；中国AI追随之路的五大误区；ECharts创始人“下海”养鱼；谷歌I/O开发者大会什么都有，只是没有惊喜
微软回应中国区AI团队“打包赴美”传闻
基于大语言模型的开源知识库问答系统 MaxKB GitHub Star 数量突破 5,000 个！
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
《2024 年一季度互联网投融资运行情况》研究报告
报告：Django 仍然是 74% 开发者的首选
周排行
返回指定时间格式
fopen函数中的mode参数
Java 单例模式探讨
Flex remoteobject工作原理探讨
寻找mplayer的便捷安装方法
30天了解30种技术系列---(26)MySQL自动化运维工具Inception
关于Jboss/Tomcat/Jetty的JNDI定义123
程序减肥，strip，eu-strip 及其符号表
AsyncTask、View.post(Runnable)、ViewTreeObserver三种方式总结frame animation自动启动
Json和Bean的互相转换
每日归档
更多
2024-05-15(24)
2024-05-14(0)
2024-05-13(18)
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022