technical background
In the last blog, we mentioned the discussion and technical implementation of the GB28181 historical video and audio file retrieval specification on the Android platform . After file retrieval, the GB28181 platform side can perform playback or download operations on the file list. This article mainly discusses the downloading of video and audio files. Related.
Standard interpretation
Basic requirements for downloading video and audio files
After receiving the video and audio file download request sent by the media receiver, the SIP server sends a media file download command to the media stream sender. The media stream sender uses RTP to transmit the video stream to the media stream receiver, and the media stream receiver directly transmits the video stream to the media stream receiver. Save as media file.
The media stream receiver can be a user client or a networked system, and the media stream sender can be a media device or a networked system. The media stream receiver or SIP server can obtain the download and send speed supported by the media stream sender through configuration query, etc., and carry the specified download speed in the requested SDP message body.
The media stream sender can expand the size parameter of the downloaded file in the 200 0K response SDP message body corresponding to the Invite request, so that the media stream receiver can calculate the download progress. When the media stream sender cannot provide the file size parameter, the media stream receiver The downloader should support calculating the download progress based on the time obtained from the code stream. Video and audio file downloads should support the media stream keep-alive mechanism.
Command process
Among them, signaling 1, 8, 9, 10, 11, 12 is the signaling process of establishing a media link between the media stream recipient and the media server through the B2BUA proxy mode after the SIP server receives the client's call request.
Signaling 2~7 is the signaling process for the SIP server to establish the media link between the media server and the media stream through three-party call control.
Signaling 13~16 is the process for the media stream sender to play back the download to the end of the file and send a download completion notification message to the media receiver.
Signaling 17~20 is the signaling process for the media stream receiver to disconnect the media link with the media server.
Signaling 21~24 is the signaling process for the SIP server to disconnect the media link between the media server and the media stream sender.
The command process is described as follows:
- The media stream receiver sends an Invite message to the SIP server. The message header field carries the Subject field, indicating the on-demand video source ID, the sender's media stream serial number, the media stream receiver ID, the receiving end's media stream serial number identification and other parameters. SDP The s field in the message body is "Download" to represent file download, the u field represents the download channel ID and download type, and the field represents the download time period. The a field can be extended to carry the download speed parameter, which specifies the download speed of the device for this download. If it is not carried The default is 1x speed.
- After receiving the Invite request, the SIP server establishes a media connection between the media server and the media stream sender through three-party call control. Send an Invite message to the media server. This message does not carry an SDP message body.
- After receiving the Invite request from the SIP server, the media server replies with a 200 0K response, carrying the SDP message body. The message body describes the IP port, media format, etc. for the media server to receive the media stream.
- After receiving the 200 OK response returned by the media server, the SIP server sends an Invite request to the media stream sender. The request carries the 200 OK response message body replied by the media server in message 3. The s field "Download" represents file download, the u field represents the download channel ID and download type, the t field represents the download time period, the y field is added to describe the SSRC value, the f field describes the media parameters, and the a field can be expanded to carry the download multiple speed, which will be doubled. Parameters passed to the device.
- After receiving the Invite request from the SIP server, the media stream sender replies with a 200 OK response, carrying the SDP message body. The message body describes the IP, port, media format, SSRC field and other contents of the media stream sent by the media stream sender, and is extensible. The a field carries the file size parameter.
- After receiving the 200 OK response from the media stream sender, the SIP server sends an ACK request to the media server. The request carries the 200 OK response message body replied by the media stream sender in message 5 to complete the Invite session establishment process with the media server.
- After receiving the 200 OK response from the media stream sender, the SIP server sends an ACK request to the media stream sender. The request does not carry the message body, and completes the Invite session establishment process with the media stream sender.
- After completing the three-party call control, the SIP server establishes a media connection between the media stream receiver and the media server through the B2BUA proxy method. Add the SSRC value to message 1 and forward it to the media server.
- The media server receives the Invite request and replies with a 200 OK response, carrying the SDP message body. The message body describes the IP, port, media format, SSRC value, etc. of the media stream sent by the media server.
- The SIP server forwards message 9 to the media stream receiver, and the extendable a field carries the file size parameter.
- After receiving the 200 OK response, the media stream receiver replies with an ACK message to complete the Invite session establishment process with the SIP server.
- The SIP server forwards message 11 to the media server to complete the Invite session establishment process with the media server.
- The media streamer sends an in-session Message after the file download is complete.
- The SIP server receives message 17 and forwards it to the media stream receiver.
- After receiving message 18, the media stream receiver replies with a 200 OK response and performs the link disconnection process.
- The SIP server forwards message 19 to the media stream sender.
- The media stream receiver sends a BYE message to the SIP server to disconnect the Invite session established with messages 1, 10, and 11 with the media stream receiver.
- After receiving the BYE message, the SIP server replies with a 200OK response and the session is disconnected.
- After receiving the BYE message, the SIP server sends a BYE message to the media server and disconnects the Invite session with the media server established by messages 8, 9, and 12.
- After receiving the BYE message, the media server replies with a 200 OK response and the session is disconnected.
- The SIP server sends a BYE message to the media server and disconnects the Invite session with the media server established by messages 2, 3, and 6.
- After receiving the BYE message, the media server replies with a 200 OK response and the session is disconnected.
- The SIP server sends a BYE message to the media stream sender and disconnects the Invite session with the media stream sender established by messages 4, 5, and 7.
- After receiving the BYE message, the media stream sender replies with a 200 OK response and the session is disconnected.
Technical realization
This article takes the Android platform GB28181 device developed by Daniu Live SDK to access side-view audio history file retrieval and download as an example (this article focuses on downloading), and introduces the relevant design ideas:
The Android device access terminal receives the INVITE SDP sent from the national standard platform:
v=0
o=34020000001380000001 0 0 IN IP4 192.168.2.154
s=Download
u=34020000001380000001:0
c=IN IP4 192.168.2.154
t=1693796426 1693796703
m=video 30002 RTP/AVP 96 97 98
a=recvonly
a=rtpmap:96 PS/90000
a=rtpmap:97 MPEG4/90000
a=rtpmap:98 H264/90000
a=downloadspeed:4
y=1200000001In the above SDP, s=Download means downloading, a=downloadspeed:4 means 4x speed download, SSRC is: 1200000001 (the first bit of SSRC is the identification bit of historical or real-time media stream, 0 is real-time video and audio, 1 is historical video and audio).
Android device access terminal replies:
v=0
o=34020000011310000039 0 0 IN IP4 192.168.2.212
s=Download
c=IN IP4 192.168.2.212
t=0 0
m=video 36576 RTP/AVP 96
a=rtpmap:96 PS/90000
a=filesize:15611511
a=sendonly
y=1200000001a=filesize:15611511 indicates that the video file size is 15611511Byte and carries the file size parameter, which is convenient for the media stream receiver to calculate the download progress (a=filesize is the size of the entire media container and the total number of bytes of the audio and video frames actually sent). .
After the national standard platform sends an Ack, it starts downloading the video and audio data. During the download process, the download speed can be adjusted through the SIP-INFO message and the MANSRTSP protocol:
PLAY RTSP/1.0
CSeq: 31129
Scale: 0.25After the access side of the Android GB28181 device completes sending the audio and video frames, the notification event type "121" is sent, indicating that the sending of historical media files is completed, and the intra-session Message is sent as follows:
<?xml version="1.0" encoding="GB2312"?>
<Notify>
<CmdType>MediaStatus</CmdType>
<SN>213466963</SN>
<DeviceID>34020000001380000001</DeviceID>
<NotifyType>121</NotifyType>
</Notify>Interface design
Signaling interface design:
/**
* Author: daniusdk.com
*/
package com.gb.ntsignalling;
public interface GBSIPAgent {
void addDownloadListener(GBSIPAgentDownloadListener downloadListener);
void removeDownloadListener(GBSIPAgentDownloadListener removeListener);
/*
*响应Invite Download 200 OK
*/
boolean respondDownloadInviteOK(long id, String deviceId, String startTime, String stopTime, MediaSessionDescription localMediaDescription);
/*
*响应Invite Download 其他状态码
*/
boolean respondDownloadInvite(int statusCode, long id, String deviceId, String startTime, String stopTime);
/*
* 媒体流发送者在文件下载结束后发Message消息通知SIP服务器回文件已发送完成
* notifyType 必须是"121“
*/
boolean notifyDownloadMediaStatus(long id, String deviceId, String startTime, String stopTime, String notifyType);
/*
*终止Download会话
*/
void terminateDownload(long id, String deviceId, String startTime, String stopTime, boolean isSendBYE);
/*
*终止所有Download会话
*/
void terminateAllDownloads(boolean isSendBYE);
}Historical video and audio download listener design:
/**
* Author: daniusdk.com
*/
package com.gb.ntsignalling;
public interface GBSIPAgentDownloadListener {
/*
*收到s=Download的文件下载Invite
*/
void ntsOnInviteDownload(long id, String deviceId, SessionDescription sessionDescription);
/*
*发送Download invite response 异常
*/
void ntsOnDownloadInviteResponseException(long id, String deviceId, String startTime, String stopTime, int statusCode, String errorInfo);
/*
* 收到CANCEL Download INVITE请求
*/
void ntsOnCancelDownload(long id, String deviceId, String startTime, String stopTime);
/*
* 收到Ack
*/
void ntsOnAckDownload(long id, String deviceId, String startTime, String stopTime);
/*
* 更改下载速度
*/
void ntsOnDownloadMANSRTSPScaleCommand(long id, String deviceId, String startTime, String stopTime, double scale);
/*
* 收到Bye
*/
void ntsOnByeDownload(long id, String deviceId, String startTime, String stopTime);
/*
* 不是在收到BYE Message情况下， 终止Download
*/
void ntsOnTerminateDownload(long id, String deviceId, String startTime, String stopTime);
/*
* Download会话对应的对话终止, 一般不会触发这个回调,目前只有在响应了200K, 但在64*T1时间后还没收到ACK,才可能会出发
收到这个, 请做相关清理处理
*/
void ntsOnDownloadDialogTerminated(long id, String deviceId, String startTime, String stopTime);
}Underlying jni interface design:
/**
* SmartPublisherJniV2.java
* Author: daniusdk.com
*/
package com.daniulive.smartpublisher;
public class SmartPublisherJniV2 {
/**
* Open publisher(启动推送实例)
*
* @param ctx: get by this.getApplicationContext()
*
* @param audio_opt:
* if 0: 不推送音频
* if 1: 推送编码前音频(PCM)
* if 2: 推送编码后音频(aac/pcma/pcmu/speex).
*
* @param video_opt:
* if 0: 不推送视频
* if 1: 推送编码前视频(NV12/I420/RGBA8888等格式)
* if 2: 推送编码后视频(AVC/HEVC)
* if 3: 层叠加模式
*
* <pre>This function must be called firstly.</pre>
*
* @return the handle of publisher instance
*/
public native long SmartPublisherOpen(Object ctx, int audio_opt, int video_opt, int width, int height);
/**
* 设置流类型
* @param type: 0:表示 live 流, 1:表示 on-demand 流, SDK默认为0(live流)
* 注意: 流类型设置当前仅对GB28181媒体流有效
* @return {0} if successful
*/
public native int SetStreamType(long handle, int type);
/**
* 投递视频 on demand包, 当前只用于GB28181推送, 注意ByteBuffer对象必须是DirectBuffer
*
* @param codec_id: 编码id, 当前支持H264和H265, 1:H264, 2:H265
*
* @param packet: 视频数据, 包格式请参考H264/H265 Annex B Byte stream format, 例如:
* 0x00000001 nal_unit 0x00000001 ...
* H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
* H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
*
* @param offset: 偏移量
* @param size: packet size
* @param pts_us: 时间戳, 单位微秒
* @param is_pts_discontinuity: 是否时间戳间断，0:未间断，1:间断
* @param is_key: 是否是关键帧, 0:非关键帧, 1:关键帧
* @param codec_specific_data: 可选参数，可传null, 对于H264关键帧包, 如果packet不含sps和pps, 可传0x00000001 sps 0x00000001 pps
* ,对于H265关键帧包, 如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps
* @param codec_specific_data_size: codec_specific_data size
* @param width: 图像宽, 可传0
* @param height: 图像高, 可传0
*
* @return {0} if successful
*/
public native int PostVideoOnDemandPacketByteBuffer(long handle, int codec_id,
ByteBuffer packet, int offset, int size, long pts_us, int is_pts_discontinuity, int is_key,
byte[] codec_specific_data, int codec_specific_data_size,
int width, int height);
/**
* 投递音频on demand包, 当前只用于GB28181推送, 注意ByteBuffer对象必须是DirectBuffer
*
* @param codec_id: 编码id, 当前支持PCMA和AAC, 65536:PCMA, 65538:AAC
* @param packet: 音频数据
* @param offset：packet偏移量
* @param size: packet size
* @param pts_us: 时间戳, 单位微秒
* @param is_pts_discontinuity: 是否时间戳间断，0:未间断，1:间断
* @param codec_specific_data: 如果是AAC的话，需要传 Audio Specific Configuration
* @param codec_specific_data_size: codec_specific_data size
* @param sample_rate: 采样率
* @param channels: 通道数
*
* @return {0} if successful
*/
public native int PostAudioOnDemandPacketByteBuffer(long handle, int codec_id,
ByteBuffer packet, int offset, int size, long pts_us, int is_pts_discontinuity,
byte[] codec_specific_data, int codec_specific_data_size,
int sample_rate, int channels);
/**
* 启动 GB28181 媒体流
*
* @return {0} if successful
*/
public native int StartGB28181MediaStream(long handle);
/**
* 停止 GB28181 媒体流
*
* @return {0} if successful
*/
public native int StopGB28181MediaStream(long handle);
/**
* 关闭推送实例，结束时必须调用close接口释放资源
*
* @return {0} if successful
*/
public native int SmartPublisherClose(long handle);
}Last processing logic
RecordDownloadListenerImpl is implemented as follows:
/**
* RecordDownloadListenerImpl.java
* Author: daniusdk.com
*/
package com.daniulive.smartpublisher;
public class RecordDownloadListenerImpl implements com.gb.ntsignalling.GBSIPAgentDownloadListener {
/*
*收到s=Download的文件下载Invite
*/
@Override
public void ntsOnInviteDownload(long id, String deviceId, SessionDescription sdp) {
if (!post_task(new OnInviteTask(this.context_, this.is_exit_, this.senders_map_, deviceId, sdp, id))) {
Log.e(TAG, "ntsOnInviteDownload post_task failed, " + RecordSender.make_print_tuple(id, deviceId, sdp.getTime().getStartTime(), sdp.getTime().getStopTime()));
// 这里不发488, 等待事务超时也可以的
GBSIPAgent agent = this.context_.get_agent();
if (agent != null)
agent.respondDownloadInvite(488, id, deviceId, sdp.getTime().getStartTime(), sdp.getTime().getStopTime());
}
}
/*
* 收到CANCEL Download INVITE请求
*/
@Override
public void ntsOnCancelDownload(long id, String deviceId, String startTime, String stopTime) {
Log.i(TAG, "ntsOnCancelDownload, " + RecordSender.make_print_tuple(id, deviceId, startTime, stopTime));
RecordSender sender = senders_map_.remove(id);
if (null == sender)
return;
StopDisposeTask task = new StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
/*
* 收到Ack
*/
@Override
public void ntsOnAckDownload(long id, String deviceId, String startTime, String stopTime) {
Log.i(TAG, "ntsOnAckDownload, "+ RecordSender.make_print_tuple(id, deviceId, startTime, stopTime));
RecordSender sender = senders_map_.get(id);
if (null == sender) {
Log.e(TAG, "ntsOnAckDownload get sender is null, " + RecordSender.make_print_tuple(id, deviceId, startTime, stopTime));
GBSIPAgent agent = this.context_.get_agent();
if (agent != null)
agent.terminateDownload(id, deviceId, startTime, stopTime, false);
return;
}
StartTask task = new StartTask(sender, this.senders_map_);
if (!post_task(task))
task.run();
}
/*
* 收到Bye
*/
@Override
public void ntsOnByeDownload(long id, String deviceId, String startTime, String stopTime) {
Log.i(TAG, "ntsOnByeDownload, "+ RecordSender.make_print_tuple(id, deviceId, startTime, stopTime));
RecordSender sender = this.senders_map_.remove(id);
if (null == sender)
return;
StopDisposeTask task = new StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
/*
* 更改下载速度
*/
@Override
public void ntsOnDownloadMANSRTSPScaleCommand(long id, String deviceId, String startTime, String stopTime, double scale) {
if (scale < 0.01) {
Log.e(TAG, "ntsOnDownloadMANSRTSPScaleCommand invalid scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId, startTime, stopTime));
return;
}
RecordSender sender = this.senders_map_.get(id);
if (null == sender) {
Log.e(TAG, "ntsOnDownloadMANSRTSPScaleCommand can not get sender, scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId, startTime, stopTime));
return;
}
sender.set_speed(scale);
Log.i(TAG, "ntsOnDownloadMANSRTSPScaleCommand, scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId, startTime, stopTime));
}
}The processing code related to file sending is as follows:
/**
* RecordSender.java
* Author: daniusdk.com
*/
package com.daniulive.smartpublisher;
public class RecordSender {
public void set_speed(double speed) {
int percent_speed = (int)(speed*100);
this.percent_speed_.set(percent_speed);
}
public void set_file_description(RecordFileDescription desc) {
this.file_description_ = desc;
}
public static String make_print_tuple(long id, String device_id, String start_time, String stop_time) {
StringBuilder sb = new StringBuilder(96);
sb.append("[id:").append(id);
sb.append(", device:" + device_id);
sb.append(", t=").append(start_time).append(" ").append(start_time);
sb.append("]");
return sb.toString();
}
public boolean start() {
SendThread current_thread = thread_.get();
if (current_thread != null) {
if (current_thread.is_exit()) {
Log.e(TAG, "start, the thread already exists and has exited, return false, " + get_print_tuple());
return false;
}
Log.i(TAG, "start, the thread already exists and has exited, return true, " + get_print_tuple());
return true;
}
SendThread thread = new SendThread();
if (!thread_.compareAndSet(null, thread)) {
Log.i(TAG, "start, call compareAndSet return false, the thread already exists, return true, " + get_print_tuple());
return true;
}
try {
Log.i(TAG, "start thread, " + get_print_tuple());
thread.start();
}catch (Exception e) {
thread_.compareAndSet(thread, null);
Log.e(TAG, "start e:", e);
return false;
}
return true;
}
public void stop() {
SendThread current_thread = thread_.get();
if (current_thread != null && !current_thread.is_exit()) {
current_thread.exit();
Log.i(TAG, "stop, exit thread " + get_print_tuple());
}
}
private boolean init_native_sender(StackDisposable disposables) {
if(native_handle_ !=0) {
Log.e(TAG, "init_native_sender, native_handle_ is not 0, " + get_print_tuple());
return false;
}
if (null == this.media_info_ || !this.media_info_.is_has_track() ) {
Log.e(TAG, "init_native_sender, there is no track, " + get_print_tuple());
return false;
}
if (0 == rtp_handle_) {
Log.e(TAG, "init_native_sender, rtp_handle_ is 0, " + get_print_tuple());
return false;
}
if (null == lib_publisher_){
Log.e(TAG, "init_native_sender, lib_publisher_ is null, " + get_print_tuple());
return false;
}
Context context = this.context_.get_context();
if (null == context) {
Log.e(TAG, "init_native_sender, context is null, " + get_print_tuple());
return false;
}
long handle = lib_publisher_.SmartPublisherOpen(context, media_info_.is_has_audio_track()?2:0, media_info_.is_has_video_track()?2:0, 0, 0);
if (0 == handle) {
Log.e(TAG, "init_native_sender, call SmartPublisherOpen failed, " + get_print_tuple());
return false;
}
NativeSenderDisposable native_disposable = new NativeSenderDisposable(lib_publisher_, handle);
lib_publisher_.SetStreamType(handle, 1);
List<MediaTrack> tracks = media_info_.get_tracks();
for (MediaTrack i : tracks) {
if (i.is_video())
lib_publisher_.SetEncodedVideoCodecId(handle, i.codec_id(), i.csd_set(), i.csd_set() != null? i.csd_set().length : 0);
else if(i.is_audio())
lib_publisher_.SetEncodedAudioCodecId(handle, i.codec_id(), i.csd_set(), i.csd_set() != null? i.csd_set().length : 0);
}
lib_publisher_.SetGB28181RTPSender(handle, rtp_handle_, rtp_payload_type_, rtp_encoding_name_);
int ret = lib_publisher_.StartGB28181MediaStream(handle);
if (ret != 0) {
Log.e(TAG, "init_native_sender, call StartGB28181MediaStream failed, " + get_print_tuple());
native_disposable.dispose();
return false;
}
native_disposable.is_need_call_stop(true);
disposables.push(native_disposable);
native_handle_ = handle;
return true;
}
private boolean post_media_packet(MediaPacket packet) {
/*Log.i(TAG, "post "+ MediaTrack.get_media_type_string(packet.media_type()) + " " +
MediaTrack.get_codec_id_string(packet.codec_id()) + " packet, pts:" + out_point_3(packet.pts_us()/1000.0) +"ms, key:"
+ (packet.is_key()?1:0) + ", size:" + packet.size()); */
if (null == lib_publisher_ || 0 == native_handle_ || !packet.is_has_data())
return false;
if (packet.is_audio()) {
if (packet.is_aac()) {
if (packet.is_has_codec_specific_data_set())
return 0 == lib_publisher_.PostAudioOnDemandPacketByteBuffer(native_handle_, packet.codec_id(), packet.data(), 0, packet.size(),
packet.pts_us(), 0, packet.codec_specific_data_set(), packet.codec_specific_data_set_size(), 0, 0);
}
}else if (packet.is_video()) {
if (packet.is_avc() || packet.is_hevc()) {
return 0 == lib_publisher_.PostVideoOnDemandPacketByteBuffer(native_handle_, packet.codec_id(), packet.data(),
0, packet.size(), packet.pts_us(), 0, packet.is_key()?1:0,
packet.codec_specific_data_set(), packet.codec_specific_data_set_size(), 0, 0);
}
}
return false;
}
private void release_packets(Deque<MediaPacket> packets) {
while (!packets.isEmpty())
packets.removeFirst().release_buffer();
}
private static String out_point_3(double v) { return String.format("%.3f", v); }
public static String to_mega_bytes_string(long bytes) {
double mb = bytes/(1024*1024.0);
return out_point_3(mb);
}
private class SendThread extends Thread {
@Override
public void run() {
/***
*相关代码
**/
}
}
}Summarize
GB28181 historical video and audio file download seems to be logically complex, but in fact it is not simple. The file download is based on completing the video and historical video and audio file retrieval, respectively from the perspectives of signaling, RTP data packaging and sending, etc., taking into account the video File integrity, the design goal of downloading historical video and audio files is to reduce frame loss and packet loss. It is recommended to use RTP over TCP mode. Although as the access side of GB28181 equipment, we try our best to implement it in accordance with standard specifications. The actual docking national standard platform manufacturer , there will be some differences, and the details will be dealt with according to the actual situation on site.