technical background
The GB28181 protocol is a protocol for reporting device status information, which can communicate and transmit data between different devices.
It is very necessary to implement GB/T 28181 on the Android system. The implementation of the GB28181 protocol is divided into two parts, one part is signaling, and the other part is the encoding of media data.
Signaling mainly includes request and response processing of SIP Register, SIP Message, SIP Invite, SIP NOTIFY, SIP SUBSCRIBE and other methods, as well as the analysis and generation of MANSCDP.
Video mainly encodes the camera image into H.264 or H.265, audio mainly encodes the audio collected by the microphone into G.711 or aac, and then packs the encoded audio and video data into PS packets, and then packs the PS packets into the RTP packet, and then send the RTP packet.
If it is self-developed, the ideas that can be used for reference are as follows: For example, use the c++ code based on esosip and osp library to develop the client of GB28181 protocol.
Then, in the Android application, it is necessary to realize the functions of decoding and rendering and playing audio and video. You can pass the Surface to the Native layer and use the ANativeWindow_fromSurface function to obtain the ANativeWindow object as a carrier for rendering and decoding data. Of course, it can also be directly collected and transmitted through NV12 or NV21 data.
In terms of signaling, it is also necessary to set an appropriate heartbeat interval and number of heartbeats to maintain a connection with the server.
It should be noted that when implementing the access of the GB28181 protocol on the Android platform, compatibility and performance issues need to be considered. In particular, for different versions of the Android operating system, corresponding compatibility processing is required. Generally speaking, considering the encoding performance, it is recommended to select a device that supports hard coding to ensure that the resolution can support up to 1920*1080 or even higher.
Many developers want to know our design ideas. Taking our Android platform GB28181 device access module as an example, our design is as follows:
Technical realization
GBSIPAgentListener is mainly related to GB28181 registration, heartbeat, DevicePosition, etc., such as successful registration, registration timeout, registration network transport layer error, abnormal heartbeat, device position request processing:
public interface GBSIPAgentListener
{
/*注册成功
* @param dateString: 服务器日期，用来校准设备端时间，用户自行决定是否校准设备时间
*/
void ntsRegisterOK(String dateString);
/*
*注册超时
*/
void ntsRegisterTimeout();
/*
*注册网络传输层异常
*/
void ntsRegisterTransportError(String errorInfo);
/*
*心跳达到异常次数
*/
void ntsOnHeartBeatException(int exceptionCount, String lastExceptionInfo);
/*
* 设备位置请求, 这个主要用在移动设备位置订阅上
* @param interval 请求间隔, 单位是毫秒
*/
void ntsOnDevicePositionRequest(String deviceId, int interval);
}GBSIPAgentPlayListener mainly deals with Invite, Ack, Bye, etc. of GB28181:
public interface GBSIPAgentPlayListener {
/*
*收到s=Play的实时视音频点播
*/
void ntsOnInvitePlay(String deviceId, SessionDescription sessionDescription);
/*
*发送play invite response 异常
*/
void ntsOnPlayInviteResponseException(String deviceId, int statusCode, String errorInfo);
/*
* 收到CANCEL play INVITE请求
*/
void ntsOnCancelPlay(String deviceId);
/*
* 收到Ack
*/
void ntsOnAckPlay(String deviceId);
/*
* 收到Bye
*/
void ntsOnByePlay(String deviceId);
/*
* 不是在收到BYE Message情况下， 终止Play
*/
void ntsOnTerminatePlay(String deviceId);
/*
* Play会话对应的对话终止, 一般不会出发这个回调,目前只有在响应了200K, 但在64*T1时间后还没收到ACK,才可能会出发
收到这个, 请做相关清理处理
*/
void ntsOnPlayDialogTerminated(String deviceId);
}GBSIPAgentAudioBroadcastListener is mainly related to GB28181 voice broadcast processing. If you have voice broadcast related needs, you can refer to the demo example to realize:
public interface GBSIPAgentAudioBroadcastListener {
/*
*收到语音广播通知
*/
void ntsOnNotifyBroadcastCommand(String fromUserName, String fromUserNameAtDomain, String sn, String sourceID, String targetID);
/*
*需要准备接受语音广播的SDP内容
*/
void ntsOnAudioBroadcast(String commandFromUserName, String commandFromUserNameAtDomain, String sourceID, String targetID);
/*
*音频广播, 发送Invite请求异常
*/
void ntsOnInviteAudioBroadcastException(String sourceID, String targetID, String errorInfo);
/*
*音频广播, 等待Invite响应超时
*/
void ntsOnInviteAudioBroadcastTimeout(String sourceID, String targetID);
/*
*音频广播, 收到Invite消息最终响应
*/
void ntsOnInviteAudioBroadcastResponse(String sourceID, String targetID, int statusCode, SessionDescription sessionDescription);
/*
* 音频广播, 收到BYE Message
*/
void ntsOnByeAudioBroadcast(String sourceID, String targetID);
/*
* 不是在收到BYE Message情况下， 终止音频广播
*/
void ntsOnTerminateAudioBroadcast(String sourceID, String targetID);
}GBSIPAgentDeviceControlListener is mainly related to GB28181 device control, such as remote start and PTZ control:
public interface GBSIPAgentDeviceControlListener {
/*
* 收到远程启动控制命令
*/
void ntsOnDeviceControlTeleBootCommand(String deviceId, String teleBootValue);
/*
* 云台控制
*/
void ntsOnDeviceControlPTZCmd(String deviceId, String typeValue);
}GBSIPAgentQueryCommandListener is mainly for GB28181 query commands, such as preset position query:
public interface GBSIPAgentQueryCommandListener {
/*
* 设备预置位查询
*/
void ntsOnDevicePresetQueryCommand(String fromUserName, String fromUserNameAtDomain, String sn, String deviceId);
}GBSIPAgentTalkListener mainly deals with GB28181 voice intercom:
public interface GBSIPAgentTalkListener {
/*
*收到s=Talk 语音对讲
*/
void ntsOnInviteTalk(String deviceId, SessionDescription sessionDescription);
/*
*发送talk invite response 异常
*/
void ntsOnTalkInviteResponseException(String deviceId, int statusCode, String errorInfo);
/*
* 收到CANCEL Talk INVITE请求
*/
void ntsOnCancelTalk(String deviceId);
/*
* 收到Ack
*/
void ntsOnAckTalk(String deviceId);
/*
* 收到Bye
*/
void ntsOnByeTalk(String deviceId);
/*
* 不是在收到BYE Message情况下， 终止Talk
*/
void ntsOnTerminateTalk(String deviceId);
/*
* Talk会话对应的对话终止, 一般不会出发这个回调,目前只有在响应了200K, 但在64*T1时间后还没收到ACK,才可能会出发
收到这个, 请做相关清理处理
*/
void ntsOnTalkDialogTerminated(String deviceId);
}media data processing
RTP data sending
RTP Sender (SmartPublisherJniV2.java) related interface design:
/*
* SmartPublisherJniV2.java
* Author: https://daniusdk.com
*/
/*
* 创建RTP Sender实例
*
* @param reserve：保留参数传0
*
* @return RTP Sender 句柄，0表示失败
*/
public native long CreateRTPSender(int reserve);
/**
*设置 RTP Sender传输协议
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param transport_protocol, 0:UDP, 1:TCP, 默认是UDP
*
* @return {0} if successful
*/
public native int SetRTPSenderTransportProtocol(long rtp_sender_handle, int transport_protocol);
/**
*设置 RTP Sender IP地址类型
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param ip_address_type, 0:IPV4, 1:IPV6, 默认是IPV4, 当前仅支持IPV4
*
* @return {0} if successful
*/
public native int SetRTPSenderIPAddressType(long rtp_sender_handle, int ip_address_type);
/**
*设置 RTP Sender RTP Socket本地端口
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param port, 必须是偶数，设置0的话SDK会自动分配, 默认值是0
*
* @return {0} if successful
*/
public native int SetRTPSenderLocalPort(long rtp_sender_handle, int port);
/**
*设置 RTP Sender SSRC
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param ssrc, 如果设置的话，这个字符串要能转换成uint32类型, 否则设置失败
*
* @return {0} if successful
*/
public native int SetRTPSenderSSRC(long rtp_sender_handle, String ssrc);
/**
*设置 RTP Sender RTP socket 发送Buffer大小
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param buffer_size, 必须大于0, 默认是512*1024, 当前仅对UDP socket有效, 根据视频码率考虑设置合适的值
*
* @return {0} if successful
*/
public native int SetRTPSenderSocketSendBuffer(long rtp_sender_handle, int buffer_size);
/**
*设置 RTP Sender RTP时间戳时钟频率
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param clock_rate, 必须大于0, 对于GB28181 PS规定是90kHz, 也就是90000
*
* @return {0} if successful
*/
public native int SetRTPSenderClockRate(long rtp_sender_handle, int clock_rate);
/**
*设置 RTP Sender 目的IP地址, 注意当前用在GB2818推送上,只设置一个地址,将来扩展如果用在其他地方，可能要设置多个目的地址，到时候接口可能会调整
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param address, IP地址
* @param port, 端口
*
* @return {0} if successful
*/
public native int SetRTPSenderDestination(long rtp_sender_handle, String address, int port);
/**
* 设置是否开启 RTP Receiver
* @param rtp_sender_handle, CreateRTPSender返回值
* @param is_enable, 0表示不收RTP包, 1表示收RTP包, SDK默认值为0.
* @return
*/
public native int EnableRTPSenderReceive(long rtp_sender_handle, int is_enable);
/**
*设置RTP Receiver SSRC
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param ssrc, 如果设置的话，这个字符串要能转换成uint32类型, 否则设置失败
*
* @return {0} if successful
*/
public native int SetRTPSenderReceiveSSRC(long rtp_sender_handle, String ssrc);
/**
*设置RTP Receiver Payload 相关信息
*
* @param rtp_sender_handle, CreateRTPSender返回值
*
* @param payload_type, 请参考 RFC 3551
*
* @param encoding_name, 编码名, 请参考 RFC 3551, 如果payload_type不是动态的, 可能传null就好
*
* @param media_type, 媒体类型, 请参考 RFC 3551, 1 是视频, 2是音频
*
* @param clock_rate, 请参考 RFC 3551
*
* @return {0} if successful
*/
public native int SetRTPSenderReceivePayloadType(long rtp_sender_handle, int payload_type, String encoding_name, int media_type, int clock_rate);
/**
*设置RTP Receiver PS的pts和dts clock frequency
*
* @param rtp_sender_handle, CreateRTPSender返回值
*
* @param ps_clock_frequency, 默认是90000, 一些特殊场景需要设置
*
* @return {0} if successful
*/
public native int SetRTPSenderReceivePSClockFrequency(long rtp_sender_handle, int ps_clock_frequency);
/**
*设置 RTP Receiver 音频采样率
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param sampling_rate, 音频采样率
*
* @return {0} if successful
*/
public native int SetRTPSenderReceiveAudioSamplingRate(long rtp_sender_handle, int sampling_rate);
/**
*设置 RTP Receiver 音频通道数
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param channels, 音频通道数
*
* @return {0} if successful
*/
public native int SetRTPSenderReceiveAudioChannels(long rtp_sender_handle, int channels);
/**
*初始化RTP Sender, 初始化之前先调用上面的接口配置相关参数
*
* @param rtp_sender_handle, CreateRTPSender返回值
*
* @return {0} if successful
*/
public native int InitRTPSender(long rtp_sender_handle);
/**
*获取RTP Sender RTP Socket本地端口
*
* @param rtp_sender_handle, CreateRTPSender返回值
*
* @return 失败返回0, 成功的话返回响应的端口, 请在InitRTPSender返回成功之后调用
*/
public native int GetRTPSenderLocalPort(long rtp_sender_handle);
/**
* UnInit RTP Sender
*
* @param rtp_sender_handle, CreateRTPSender返回值
*
* @return {0} if successful
*/
public native int UnInitRTPSender(long rtp_sender_handle);
/**
* 释放RTP Sender, 释放之后rtp_sender_handle就无效了，请不要再使用
*
* @param rtp_sender_handle, CreateRTPSender返回值
*
* @return {0} if successful
*/
public native int DestoryRTPSender(long rtp_sender_handle);RTP data reception
Corresponding to the RTP Receiver (SmartPlayerJniV2.java) related interface design, if there is no technical requirement for voice broadcast or voice intercom, this part can be ignored:
/*
* SmartPlayerJniV2.java
* Author: https://daniusdk.com
*/
/*
* 创建RTP Receiver
*
* @param reserve：保留参数传0
*
* @return RTP Receiver 句柄，0表示失败
*/
public native long CreateRTPReceiver(int reserve);
/**
*设置 RTP Receiver传输协议
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param transport_protocol, 0:UDP, 1:TCP, 默认是UDP
*
* @return {0} if successful
*/
public native int SetRTPReceiverTransportProtocol(long rtp_receiver_handle, int transport_protocol);
/**
*设置 RTP Receiver IP地址类型
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param ip_address_type, 0:IPV4, 1:IPV6, 默认是IPV4
*
* @return {0} if successful
*/
public native int SetRTPReceiverIPAddressType(long rtp_receiver_handle, int ip_address_type);
/**
*设置 RTP Receiver RTP Socket本地端口
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param port, 必须是偶数，设置0的话SDK会自动分配, 默认值是0
*
* @return {0} if successful
*/
public native int SetRTPReceiverLocalPort(long rtp_receiver_handle, int port);
/**
*设置 RTP Receiver SSRC
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param ssrc, 如果设置的话，这个字符串要能转换成uint32类型, 否则设置失败
*
* @return {0} if successful
*/
public native int SetRTPReceiverSSRC(long rtp_receiver_handle, String ssrc);
/**
*创建 RTP Receiver 会话
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param reserve, 保留值，目前传0
*
* @return {0} if successful
*/
public native int CreateRTPReceiverSession(long rtp_receiver_handle, int reserve);
/**
*获取 RTP Receiver RTP Socket本地端口
*
* @param rtp_receiver_handle, CreateRTPReceiver
*
* @return 失败返回0, 成功的话返回响应的端口, 请在CreateRTPReceiverSession返回成功之后调用
*/
public native int GetRTPReceiverLocalPort(long rtp_receiver_handle);
/**
*设置 RTP Receiver Payload 相关信息
*
* @param rtp_receiver_handle, CreateRTPReceiver
*
* @param payload_type, 请参考 RFC 3551
*
* @param encoding_name, 编码名, 请参考 RFC 3551, 如果payload_type不是动态的, 可能传null就好
*
* @param media_type, 媒体类型, 请参考 RFC 3551, 1 是视频, 2是音频
*
* @param clock_rate, 请参考 RFC 3551
*
* @return {0} if successful
*/
public native int SetRTPReceiverPayloadType(long rtp_receiver_handle, int payload_type, String encoding_name, int media_type, int clock_rate);
/**
*设置 RTP Receiver 音频采样率
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param sampling_rate, 音频采样率
*
* @return {0} if successful
*/
public native int SetRTPReceiverAudioSamplingRate(long rtp_receiver_handle, int sampling_rate);
/**
*设置 RTP Receiver 音频通道数
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param channels, 音频通道数
*
* @return {0} if successful
*/
public native int SetRTPReceiverAudioChannels(long rtp_receiver_handle, int channels);
/**
*设置 RTP Receiver 远端地址
*
* @param rtp_receiver_handle, CreateRTPReceiver
* @param address, IP地址
* @param port, 端口
*
* @return {0} if successful
*/
public native int SetRTPReceiverRemoteAddress(long rtp_receiver_handle, String address, int port);
/**
*初始化 RTP Receiver
*
* @param rtp_receiver_handle, CreateRTPReceiver
*
* @return {0} if successful
*/
public native int InitRTPReceiver(long rtp_receiver_handle);
/**
*UnInit RTP Receiver
*
* @param rtp_receiver_handle, CreateRTPReceiver
*
* @return {0} if successful
*/
public native int UnInitRTPReceiver(long rtp_receiver_handle);
/**
*Destory RTP Receiver Session
*
* @param rtp_receiver_handle, CreateRTPReceiver
*
* @return {0} if successful
*/
public native int DestoryRTPReceiverSession(long rtp_receiver_handle);
/**
*Destory RTP Receiver
*
* @param rtp_receiver_handle, CreateRTPReceiver
*
* @return {0} if successful
*/
public native int DestoryRTPReceiver(long rtp_receiver_handle);PostAudioPacket (SmartPlayerJniV2.java), deliver audio packets to external Live source, currently only used for voice intercom:
/*
* SmartPlayerJniV2.java
* Author: https://daniusdk.com
*/
/**
* 投递音频包给外部Live source, 注意ByteBuffer对象必须是DirectBuffer
*
* @param handle: return value from SmartPlayerOpen()
*
* @return {0} if successful
*/
public native int PostAudioPacket(long handle, int codec_id,
java.nio.ByteBuffer packet, int offset, int size, long pts, boolean is_pts_discontinuity,
java.nio.ByteBuffer extra_data, int extra_data_offset, int extra_data_size, int sample_rate, int channels);GB28181 interface call
Corresponding to GB28181 related interface call related design is as follows:
/*
* SmartPublisherJniV2.java
* Author: https://daniusdk.com
*/
/**
* 设置GB28181 RTP Sender
*
* @param rtp_sender_handle, CreateRTPSender返回值
* @param rtp_payload_type, 对于GB28181 PS, 协议定义是96, 具体以SDP为准, RFC 3551有定义
* @param encoding_name, 编码名, 请参考 RFC 3551, 当前仅支持: "PS", 其他值返回失败
* @return {0} if successful
*/
public native int SetGB28181RTPSender(long handle, long rtp_sender_handle, int rtp_payload_type, String encoding_name);
/**
* 设置GB28181 RTP 收到的音频包回调
* @param handle
* @param audio_packet_callback
* @return
*/
public native int SetGB28181ReceiveAudioPacketCallback(long handle, NTAudioPacketCallback audio_packet_callback);
/**
* 启动 GB28181 媒体流
*
* @return {0} if successful
*/
public native int StartGB28181MediaStream(long handle);
/**
* 停止 GB28181 媒体流
*
* @return {0} if successful
*/
public native int StopGB28181MediaStream(long handle);Summarize
In addition to the above design, the module can also be extended to realize real-time mute, real-time snapshot, on-demand video recording, real-time volume adjustment, etc., to realize customized technical demands.