3. Buffer has no PTS and runtime check failed

1 Source of problem

1.1 Buffer has no PTS

When I use mp4mux or qtmux, the problem of Buffer has no PTS occasionally occurs.

PTS (timestamp) is an important parameter used to identify the position of audio and video data on the timeline. The lack of timestamp information will result in the inability to determine the position of the data on the timeline, thus preventing correct encapsulation and processing.

In GStreamer, DTS stands for "Decode Time Stamp". DTS is a timestamp used for video and audio data streams that represents the decoding time of each sample in the media data stream. Each sample is associated with a timestamp, which represents the time in the media clock that the sample should play.

DTS Typically used to synchronize audio and video as it tells the player when audio or video samples should be decoded and presented to the user. In video, DTS represents how long each frame of video should be rendered after decoding. In audio, DTS represents when each audio sample should be played. Typically, a media decoder decodes media samples against the DTS to ensure that the samples are played at the correct time.

It should be noted that DTS is different from PTS (Presentation Time Stamp), PTS 表示样本应该在播放时呈现的时间, and DTS 则表示样本应该在解码时呈现的时间. PTS is often used to synchronize audio and video and deal with latency issues.

When using DeepStream, when parsing the rtsp stream of Hikvision camera
PTS is sometimes invalid
DTS = -1
Duration = -1

1.2 runtime check failed

It may also be because of my GStreamer version. The following warning comes from NVIDIA's H264 codec plug-in (nvv4l2decoder, nvv4l2h264enc). It is currently unclear how this warning is related to PTS, and the details need to be tested.

(vpf-app:45070): GStreamer-WARNING **: 01:26:03.100: (../subprojects/gstreamer/gst/gstinfo.c:653):gst_debug_log_valist: runtime check failed: (object == NULL || G_IS_OBJECT (object))

(vpf-app:45070): GStreamer-WARNING **: 01:26:03.101: (../subprojects/gstreamer/gst/gstinfo.c:653):gst_debug_log_valist: runtime check failed: (object == NULL || G_IS_OBJECT (object))

(vpf-app:45070): GStreamer-WARNING **: 01:26:03.101: (../subprojects/gstreamer/gst/gstinfo.c:653):gst_debug_log_valist: runtime check failed: (object == NULL || G_IS_OBJECT (object))

(vpf-app:45070): GStreamer-WARNING **: 01:26:03.101: (../subprojects/gstreamer/gst/gstinfo.c:653):gst_debug_log_valist: runtime check failed: (object == NULL || G_IS_OBJECT (object))

Because no specific impact was found, the plug-in is not open source, and related functions are temporarily disabled in the GStreamer source program.

Please add image description

2 Solutions found online

2.1 OpenAI provides several solutions

The solution to this problem is to add timestamp information to the live stream. If you are using GStreamer for development, you can use the corresponding plug-in to add timestamp information, such as using the "timestamp-mode" attribute in the "identity" plug-in or the "videotestsrc" plug-in to add timestamp information. Alternatively, you can use the "queue2" plugin to cache the data and ensure that the timestamp information is delivered correctly.

If your live stream does not have timestamp information, you will need to add the timestamp information before decoding the data. The specific method may vary depending on the data source, but generally speaking, you can use the current system time or the sampling time of the audio and video frame as the timestamp information.

It should be noted that the processing and encapsulation of real-time streams are relatively complex, and factors such as data transmission delay, frame rate, and code rate need to be reasonably set and optimized to ensure correct decoding and encapsulation of data.

GStreamer mp4mux gives “Buffer has no PTS” error using custom appsrc by predicting timestamps (文中的方法不稳定，元素运行时间与PTS时间存在变化，没有确定关系).
Gstreamer for Android Buffer has no PTS Set the timestamp inference mode and enable interpolation PTS processing through h264parse. (好像只有缺失B帧的情况才有用).

  gst_base_parse_set_infer_ts (GST_BASE_PARSE (bin->codecparse), TRUE);
  gst_base_parse_set_pts_interpolation (GST_BASE_PARSE (bin->codecparse), TRUE);

3. Both mp4mux and qtmux will have Buffer missing PTS, but it does not appear when using Matroska media container (成功)

mp4mux, qtmux, and matroskamux are all components of media container formats used to package audio and video data into a single file. Their differences are as follows:
mp4mux: Used to package audio and video streams into the MP4 file format. It is commonly used on mobile devices and streaming services.
qtmux: Used to package audio and video streams into QuickTime file format. It is commonly used in Apple devices and software.
matroskamux: Used to package audio and video streams into the Matroska file format, also known as MKV. It is commonly used in video compression and digital storage devices.
Overall, the main difference between these components is the container formats they support. Each container format has its own characteristics and applicable scenarios, and needs to be selected according to specific usage requirements.