technical background
When we connected the GB28181 device access module, we encountered such technical demands. Many developers expected to provide encoded (H.264/H.265, AAC/PCMA) data connection to ensure external collection equipment, such as unmanned aerial vehicle The data called back by the machine is directly connected to the GB28181 platform side through the module. In addition, it is also expected that RTSP devices that do not support or the internal network does not have external network permissions can also be indirectly connected to the national standard platform.
Technical realization
Encoded audio and video data
This article takes the Android platform as an example. Based on the above requirements, we designed the interface as follows. Simply put, the GB28181 interaction process remains unchanged, as long as the data access interface is provided:
/**
* 设置编码后视频数据(H.264)
*
* @param codec_id, H.264对应 1
*
* @param data 编码后的video数据
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
*
* @param timestamp video timestamp
*
* @param pts Presentation Time Stamp, 显示时间戳
*
* @return {0} if successful
*/
public native int SmartPublisherPostVideoEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp, long pts);
/**
* 设置编码后视频数据(H.264)
*
* @param codec_id, H.264对应 1
*
* @param data 编码后的video数据
*
*@param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
*
* @param timestamp video timestamp
*
* @param pts Presentation Time Stamp, 显示时间戳
*
* @return {0} if successful
*/
public native int SmartPublisherPostVideoEncodedDataV2(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp, long pts,
byte[] sps, int sps_len,
byte[] pps, int pps_len);
/**
* 设置编码后视频数据(H.264)，如需录制编码后的数据，用此接口，且设置实际宽高
*
* @param codec_id, H.264对应 1
*
* @param data 编码后的video数据
*
*@param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
*
* @param timestamp video timestamp
*
* @param pts Presentation Time Stamp, 显示时间戳
*
* @param width, height: 编码后视频宽高
*
* @return {0} if successful
*/
public native int SmartPublisherPostVideoEncodedDataV3(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp, long pts,
byte[] sps, int sps_len,
byte[] pps, int pps_len,
int width, int height);
/**
* 设置音频数据(AAC/PCMA/PCMU/SPEEX)
*
* @param codec_id:
*
* NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
* NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
* NT_MEDIA_CODEC_ID_PCMU,
* NT_MEDIA_CODEC_ID_AAC,
* NT_MEDIA_CODEC_ID_SPEEX,
* NT_MEDIA_CODEC_ID_SPEEX_NB,
* NT_MEDIA_CODEC_ID_SPEEX_WB,
* NT_MEDIA_CODEC_ID_SPEEX_UWB,
*
* @param data audio数据
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
*
* @param timestamp video timestamp
*
* @param parameter_info 用于AAC special config信息填充
*
* @param parameter_info_size parameter info size
*
* @return {0} if successful
*/
public native int SmartPublisherPostAudioEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp,ByteBuffer parameter_info, int parameter_info_size);
/**
* 设置音频数据(AAC/PCMA/PCMU/SPEEX)
*
* @param codec_id:
*
* NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
* NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
* NT_MEDIA_CODEC_ID_PCMU,
* NT_MEDIA_CODEC_ID_AAC,
* NT_MEDIA_CODEC_ID_SPEEX,
* NT_MEDIA_CODEC_ID_SPEEX_NB,
* NT_MEDIA_CODEC_ID_SPEEX_WB,
* NT_MEDIA_CODEC_ID_SPEEX_UWB,
*
* @param data audio数据
*
* @param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
*
* @param timestamp video timestamp
*
* @param parameter_info 用于AAC special config信息填充
*
* @param parameter_info_size parameter info size
*
* @return {0} if successful
*/
public native int SmartPublisherPostAudioEncodedDataV2(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp,
byte[] parameter_info, int parameter_info_size);
/**
* 设置音频数据(AAC/PCMA/PCMU/SPEEX)
*
* @param codec_id:
*
* NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
* NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
* NT_MEDIA_CODEC_ID_PCMU,
* NT_MEDIA_CODEC_ID_AAC,
* NT_MEDIA_CODEC_ID_SPEEX,
* NT_MEDIA_CODEC_ID_SPEEX_NB,
* NT_MEDIA_CODEC_ID_SPEEX_WB,
* NT_MEDIA_CODEC_ID_SPEEX_UWB,
*
* @param data audio数据
*
* @param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
*
* @param timestamp video timestamp
*
* @param parameter_info 用于AAC special config信息填充
*
* @param parameter_info_size parameter info size
*
* @param sample_rate 采样率,如果需要录像的话必须传正确的值
*
*@param channels 通道数, 如果需要录像的话必须传正确的值, 一般是1或者2
*
* @return {0} if successful
*/
public native int SmartPublisherPostAudioEncodedDataV3(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp,
byte[] parameter_info, int parameter_info_size,
int sample_rate, int channels);Pull the RTSP stream and connect it to the GB28181 platform
To put it simply, pull down the RTSP stream data of the camera, and then call back the encoded data to the upper layer. The upper layer implements PS packaging according to the GB28181 data format requirements, and then interacts with the GB28181 platform signaling and data. The national standard platform side needs to be previewed. When, after signaling interaction, just pull RTSP.
How to pull streams:
private boolean StartPull()
{
if ( isPulling )
return false;
if (!OpenPullHandle())
return false;
libPlayer.SmartPlayerSetAudioDataCallback(playerHandle, new PlayerAudioDataCallback());
libPlayer.SmartPlayerSetVideoDataCallback(playerHandle, new PlayerVideoDataCallback());
int is_pull_trans_code = 1;
libPlayer.SmartPlayerSetPullStreamAudioTranscodeAAC(playerHandle, is_pull_trans_code);
int startRet = libPlayer.SmartPlayerStartPullStream(playerHandle);
if (startRet != 0) {
Log.e(TAG, "Failed to start pull stream!");
if(!isPlaying && !isRecording && isPushing && !isRTSPPublisherRunning)
{
libPlayer.SmartPlayerClose(playerHandle);
playerHandle = 0;
}
return false;
}
isPulling = true;
return true;
}
private void StopPull()
{
if ( !isPulling )
return;
libPlayer.SmartPlayerStopPullStream(playerHandle);
if ( !isPlaying && !isRecording && !isPushing && !isRTSPPublisherRunning)
{
libPlayer.SmartPlayerClose(playerHandle);
playerHandle = 0;
}
isPulling = false;
}The pulled audio and video data is delivered to the GB28181 access module:
class PlayerAudioDataCallback implements NTAudioDataCallback
{
private int audio_buffer_size = 0;
private int param_info_size = 0;
private ByteBuffer audio_buffer_ = null;
private ByteBuffer parameter_info_ = null;
@Override
public ByteBuffer getAudioByteBuffer(int size)
{
//Log.i("getAudioByteBuffer", "size: " + size);
if( size < 1 )
{
return null;
}
if ( size <= audio_buffer_size && audio_buffer_ != null )
{
return audio_buffer_;
}
audio_buffer_size = size + 512;
audio_buffer_size = (audio_buffer_size+0xf) & (~0xf);
audio_buffer_ = ByteBuffer.allocateDirect(audio_buffer_size);
// Log.i("getAudioByteBuffer", "size: " + size + " buffer_size:" + audio_buffer_size);
return audio_buffer_;
}
@Override
public ByteBuffer getAudioParameterInfo(int size)
{
//Log.i("getAudioParameterInfo", "size: " + size);
if(size < 1)
{
return null;
}
if ( size <= param_info_size && parameter_info_ != null )
{
return parameter_info_;
}
param_info_size = size + 32;
param_info_size = (param_info_size+0xf) & (~0xf);
parameter_info_ = ByteBuffer.allocateDirect(param_info_size);
//Log.i("getAudioParameterInfo", "size: " + size + " buffer_size:" + param_info_size);
return parameter_info_;
}
public void onAudioDataCallback(int ret, int audio_codec_id, int sample_size, int is_key_frame, long timestamp, int sample_rate, int channel, int parameter_info_size, long reserve)
{
if ( audio_buffer_ == null)
return;
audio_buffer_.rewind();
if ( ret == 0 && (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning)) {
libPublisher.SmartPublisherPostAudioEncodedData(publisherHandle, audio_codec_id, audio_buffer_, sample_size, is_key_frame, timestamp, parameter_info_, parameter_info_size);
}
}
}
class PlayerVideoDataCallback implements NTVideoDataCallback
{
private int video_buffer_size = 0;
private ByteBuffer video_buffer_ = null;
@Override
public ByteBuffer getVideoByteBuffer(int size)
{
if( size < 1 )
{
return null;
}
if ( size <= video_buffer_size && video_buffer_ != null )
{
return video_buffer_;
}
video_buffer_size = size + 1024;
video_buffer_size = (video_buffer_size+0xf) & (~0xf);
video_buffer_ = ByteBuffer.allocateDirect(video_buffer_size);
return video_buffer_;
}
public void onVideoDataCallback(int ret, int video_codec_id, int sample_size, int is_key_frame, long timestamp, int width, int height, long presentation_timestamp)
{
if ( video_buffer_ == null)
return;
video_buffer_.rewind();
if ( ret == 0 && (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning) ) {
libPublisher.SmartPublisherPostVideoEncodedData(publisherHandle, video_codec_id, video_buffer_, sample_size, is_key_frame, timestamp, presentation_timestamp);
}
}
}How to preview and play external audio and video data?
In addition to transferring the encoded audio and video data to GB28181, in some scenarios, local preview or even secondary processing of the data (video analysis, real-time watermark character overlay, etc., and then secondary encoding) is required. Based on such scenario demands , we implemented the real-time preview and playback module of external encoding data on the Android platform.
The external (H.264/H.265) delivery interface is designed as follows:
// SmartPlayerJniV2.java
// Author: daniusdk.com
/**
* 投递视频包给外部Live Source
*
* @param codec_id: 编码id, 当前仅支持H264和H265, 1:H264, 2:H265
*
* @param packet: 视频数据, ByteBuffer必须是DirectBuffer, 包格式请参考H264/H265 Annex B Byte stream format, 例如:
* 0x00000001 nal_unit 0x00000001 ...
* H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
* H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
*
* @param offset: 偏移量
* @param size: packet size
* @param timestamp_ms: 时间戳, 单位毫秒
* @param is_timestamp_discontinuity: 是否时间戳间断，0:未间断，1:间断
* @param is_key: 是否是关键帧, 0:非关键帧, 1:关键帧
* @param extra_data: 可选参数，可传null, 对于H264关键帧包, 如果packet不含sps和pps, 可传0x00000001 sps 0x00000001 pps
* ,对于H265关键帧包, 如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps
* @param extra_data_size: extra_data size
* @param width: 图像宽, 可传0
* @param height: 图像高, 可传0
*
* @return {0} if successful
*/
public native int PostVideoPacketByteBuffer(long handle, int codec_id,
java.nio.ByteBuffer packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key,
byte[] extra_data, int extra_data_size, int width, int height);
/*
* 请参考 PostVideoPacketByteBuffer说明
*/
public native int PostVideoPacketByteArray(long handle, int codec_id,
byte[] packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key,
byte[] extra_data, int extra_data_size, int width, int height);The interface designs of PostVideoPacketByteBuffer() and PostVideoPacketByteArray() are basically similar. The only difference is that one data type is ByteBuffer and the other is a byte array.
Among them, codec_id is the code id. Currently, only H.264 and H.265 types are supported.
packet video data. It should be noted that the ByteBuffer must be DirectBuffer. For the packet format, please refer to H264/H265 Annex B Byte stream format, for example:
0x00000001 nal_unit 0x00000001 ...
H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
extra_data: Optional parameter, null can be passed. For H264 key frame package, if the packet does not contain sps and pps, 0x00000001 sps 0x00000001 pps can be passed. For H265 key frame package, if the packet does not contain vps, sps and pps, 0x00000001 can be passed vps 0x00000001 sps 0x00000001 pps
The audio (AAC/PCMA/PCMU) delivery interface is designed as follows:
/**
* 投递音频包给外部Live source, 注意ByteBuffer对象必须是DirectBuffer
*
* @param handle: return value from SmartPlayerOpen()
*
* @param codec_id: 编码id, 当前支持PCMA、PCMU和AAC, 65536:PCMA, 65537:PCMU, 65538:AAC
* @param packet: 音频数据
* @param offset：packet偏移量
* @param size: packet size
* @param pts_ms: 时间戳, 单位毫秒
* @param is_pts_discontinuity: 是否时间戳间断，false:未间断，true:间断
* @param extra_data: 如果是AAC的话，需要传 Audio Specific Configuration
* @param extra_data_offset: extra_data 偏移量
* @param extra_data_size: extra_data size
* @param sample_rate: 采样率
* @param channels: 通道数
*
* @return {0} if successful
*/
public native int PostAudioPacket(long handle, int codec_id,
java.nio.ByteBuffer packet, int offset, int size, long pts_ms, boolean is_pts_discontinuity,
java.nio.ByteBuffer extra_data, int extra_data_offset, int extra_data_size, int sample_rate, int channels);
/*
* 投递音频包给外部Live source, byte数组版本, 具体请参考PostAudioPacket
*
* @param is_pts_discontinuity: 是否时间戳间断，0:未间断，1:间断
* @return {0} if successful
*/
public native int PostAudioPacketByteArray(long handle, int codec_id,
byte[] packet, int offset, int size, long pts_ms, int is_pts_discontinuity,
byte[] extra_data, int extra_data_size, int sample_rate, int channels);Summarize
From the above description, you can see that GB/T 28181 audio and video data source access, whether it is pre-encoding or post-encoding data, or external RTSP stream data, including data preview, is not that easy to implement if you have the technical accumulation. Trouble, interested developers can try it.