technical background
When we are working on law enforcement recorders or command systems, we will encounter such a situation. In most scenarios, we do not need to transmit the data on the device side to the national standard platform in real time. By default, only local video recording is required. If the command center When you need to view the real-time data of the front-end device, initiate a video playback request, and then push the data to the platform side on the device side. If you need voice broadcast, just initiate a voice broadcast (broadcast), the GB28181 device access side responds, and then sends an INVITE request, etc., complete Voice broadcast and voice intercom. In addition, considering the upstream bandwidth bottleneck on the device side, generally speaking, local recording needs to be as clear as possible (such as 1920*1080 resolution), upload video data, and transmit 1280*720 resolution, which is what we traditionally mentioned. Stream encoding.
Technical realization
With these problems in mind, taking the Android platform device access module as an example, let’s analyze and solve them one by one:
code on demand
On-demand coding, only the Android platform GB28181 device access terminal is required to complete the registration of the device to the platform (register), and then the platform side initiates a catalog to query the device and maintain heartbeat information. If the real-time location information is subscribed, the real-time Just report the location information.
We only encode audio and video data when video recording is required or when the command center needs to play real-time audio and video data of the front-end equipment, so as to ensure that the resource occupation is minimized during standby.
Take the above picture as an example, just click "Start GB28181", the corresponding code is implemented as follows:
class ButtonGB28181AgentListener implements View.OnClickListener {
public void onClick(View v) {
stopAudioPlayer();
destoryRTPReceiver();
gb_broadcast_source_id_ = null;
gb_broadcast_target_id_ = null;
btnGB28181AudioBroadcast.setText("GB28181语音广播");
btnGB28181AudioBroadcast.setEnabled(false);
stopGB28181Stream();
destoryRTPSender();
if (null == gb28181_agent_ ) {
if( !initGB28181Agent() )
return;
}
if (gb28181_agent_.isRunning()) {
gb28181_agent_.terminateAllPlays(true);// 目前测试下来,发送BYE之后,有些服务器会立即发送INVITE,是否发送BYE根据实际情况看
gb28181_agent_.stop();
btnGB28181Agent.setText("启动GB28181");
}
else {
if ( gb28181_agent_.start() ) {
btnGB28181Agent.setText("停止GB28181");
}
}
}
}
initGB28181Agent() is implemented as follows:
private boolean initGB28181Agent() {
if ( gb28181_agent_ != null )
return true;
getLocation(context_);
String local_ip_addr = IPAddrUtils.getIpAddress(context_);
Log.i(TAG, "initGB28181Agent local ip addr: " + local_ip_addr);
if ( local_ip_addr == null || local_ip_addr.isEmpty() ) {
Log.e(TAG, "initGB28181Agent local ip is empty");
return false;
}
gb28181_agent_ = GBSIPAgentFactory.getInstance().create();
if ( gb28181_agent_ == null ) {
Log.e(TAG, "initGB28181Agent create agent failed");
return false;
}
gb28181_agent_.addListener(this);
gb28181_agent_.addPlayListener(this);
gb28181_agent_.addAudioBroadcastListener(this);
gb28181_agent_.addDeviceControlListener(this);
gb28181_agent_.addQueryCommandListener(this);
// 必填信息
gb28181_agent_.setLocalAddress(local_ip_addr);
gb28181_agent_.setServerParameter(gb28181_sip_server_addr_, gb28181_sip_server_port_, gb28181_sip_server_id_, gb28181_sip_domain_);
gb28181_agent_.setUserInfo(gb28181_sip_username_, gb28181_sip_password_);
//gb28181_agent_.setUserInfo(gb28181_sip_username_, gb28181_sip_username_, gb28181_sip_password_);
// 可选参数
gb28181_agent_.setUserAgent(gb28181_sip_user_agent_filed_);
gb28181_agent_.setTransportProtocol(gb28181_sip_trans_protocol_==0?"UDP":"TCP");
// GB28181配置
gb28181_agent_.config(gb28181_reg_expired_, gb28181_heartbeat_interval_, gb28181_heartbeat_count_);
com.gb.ntsignalling.Device gb_device = new com.gb.ntsignalling.Device("34020000001380000001", "安卓测试设备", Build.MANUFACTURER, Build.MODEL,
"宇宙","火星1","火星", true);
if (mLongitude != null && mLatitude != null) {
com.gb.ntsignalling.DevicePosition device_pos = new com.gb.ntsignalling.DevicePosition();
device_pos.setTime(mLocationTime);
device_pos.setLongitude(mLongitude);
device_pos.setLatitude(mLatitude);
gb_device.setPosition(device_pos);
gb_device.setSupportMobilePosition(true); // 设置支持移动位置上报
}
gb28181_agent_.addDevice(gb_device);
if (!gb28181_agent_.createSipStack()) {
gb28181_agent_ = null;
Log.e(TAG, "initGB28181Agent gb28181_agent_.createSipStack failed.");
return false;
}
boolean is_bind_local_port_ok = false;
// 最多尝试5000个端口
int try_end_port = gb28181_sip_local_port_base_ + 5000;
try_end_port = try_end_port > 65536 ?65536: try_end_port;
for (int i = gb28181_sip_local_port_base_; i < try_end_port; ++i) {
if (gb28181_agent_.bindLocalPort(i)) {
is_bind_local_port_ok = true;
break;
}
}
if (!is_bind_local_port_ok) {
gb28181_agent_.releaseSipStack();
gb28181_agent_ = null;
Log.e(TAG, "initGB28181Agent gb28181_agent_.bindLocalPort failed.");
return false;
}
if (!gb28181_agent_.initialize()) {
gb28181_agent_.unBindLocalPort();
gb28181_agent_.releaseSipStack();
gb28181_agent_ = null;
Log.e(TAG, "initGB28181Agent gb28181_agent_.initialize failed.");
return false;
}
return true;
}Registration and heartbeat exceptions are handled as follows:
@Override
public void ntsRegisterOK(String dateString) {
Log.i(TAG, "ntsRegisterOK Date: " + (dateString!= null? dateString : ""));
}
@Override
public void ntsRegisterTimeout() {
Log.e(TAG, "ntsRegisterTimeout");
}
@Override
public void ntsRegisterTransportError(String errorInfo) {
Log.e(TAG, "ntsRegisterTransportError error:" + (errorInfo != null?errorInfo :""));
}
@Override
public void ntsOnHeartBeatException(int exceptionCount, String lastExceptionInfo) {
Log.e(TAG, "ntsOnHeartBeatException heart beat timeout count reached, count:" + exceptionCount+
", exception info:" + (lastExceptionInfo!=null?lastExceptionInfo:""));
// 停止信令, 然后重启
handler_.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG, "gb28281_heart_beart_timeout");
stopAudioPlayer();
destoryRTPReceiver();
if (gb_broadcast_source_id_ != null && gb_broadcast_target_id_ != null && gb28181_agent_ != null)
gb28181_agent_.byeAudioBroadcast(gb_broadcast_source_id_, gb_broadcast_target_id_);
gb_broadcast_source_id_ = null;
gb_broadcast_target_id_ = null;
btnGB28181AudioBroadcast.setText("GB28181语音广播");
btnGB28181AudioBroadcast.setEnabled(false);
stopGB28181Stream();
destoryRTPSender();
if (gb28181_agent_ != null) {
gb28181_agent_.terminateAllPlays(true);
Log.i(TAG, "gb28281_heart_beart_timeout sip stop");
gb28181_agent_.stop();
String local_ip_addr = IPAddrUtils.getIpAddress(context_);
if (local_ip_addr != null && !local_ip_addr.isEmpty() ) {
Log.i(TAG, "gb28281_heart_beart_timeout get local ip addr: " + local_ip_addr);
gb28181_agent_.setLocalAddress(local_ip_addr);
}
Log.i(TAG, "gb28281_heart_beart_timeout sip start");
gb28181_agent_.start();
}
}
},0);
}ntsOnAckPlay(), we start encoding:
@Override
public void ntsOnAckPlay(String deviceId) {
handler_.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG,"ntsOnACKPlay, device_id:" +device_id_);
if (!isRTSPPublisherRunning && !isPushingRtmp && !isRecording) {
InitAndSetConfig();
}
libPublisher.SetGB28181RTPSender(publisherHandle, gb28181_rtp_sender_handle_, gb28181_rtp_payload_type_, gb28181_rtp_encoding_name_);
//libPublisher.SetGBTCPConnectTimeout(publisherHandle, 10*60*1000);
//libPublisher.SetGBInitialTCPReconnectInterval(publisherHandle, 1000);
//libPublisher.SetGBInitialTCPMaxReconnectAttempts(publisherHandle, 3);
int startRet = libPublisher.StartGB28181MediaStream(publisherHandle);
if (startRet != 0) {
if (!isRTSPPublisherRunning && !isPushingRtmp && !isRecording) {
if (publisherHandle != 0) {
long handle = publisherHandle;
publisherHandle = 0;
libPublisher.SmartPublisherClose(handle);
}
}
destoryRTPSender();
Log.e(TAG, "Failed to start GB28181 service..");
return;
}
if (!isRTSPPublisherRunning && !isPushingRtmp && !isRecording) {
CheckInitAudioRecorder();
}
startLayerPostThread();
isGB28181StreamRunning = true;
}
private String device_id_;
public Runnable set(String device_id) {
this.device_id_ = device_id;
return this;
}
}.set(deviceId),0);
}Among them, InitAndSetConfig() completes the basic parameter setting, such as data source type, soft and hard coding settings, frame rate key frame interval code rate and other parameter settings:
private void InitAndSetConfig() {
int audio_opt = 1;
int fps = 18;
int gop = fps * 2;
Log.i(TAG, "InitAndSetConfig video_width: " + video_width_ + " cur_video_height" + video_height_ + " imageRotationDegree:" + cameraImageRotationDegree_);
publisherHandle = libPublisher.SmartPublisherOpen(context_, audio_opt, 3, video_width_, video_height_);
if (publisherHandle == 0) {
Log.e(TAG, "sdk open failed!");
return;
}
Log.i(TAG, "publisherHandle=" + publisherHandle);
if(videoEncodeType == 1) {
int h264HWKbps = setHardwareEncoderKbps(true, video_width_, video_height_);
h264HWKbps = h264HWKbps*fps/25;
Log.i(TAG, "h264HWKbps: " + h264HWKbps);
int isSupportH264HWEncoder = libPublisher
.SetSmartPublisherVideoHWEncoder(publisherHandle, h264HWKbps);
if (isSupportH264HWEncoder == 0) {
libPublisher.SetNativeMediaNDK(publisherHandle, 0);
libPublisher.SetVideoHWEncoderBitrateMode(publisherHandle, 1); // 0:CQ, 1:VBR, 2:CBR
libPublisher.SetVideoHWEncoderQuality(publisherHandle, 39);
libPublisher.SetAVCHWEncoderProfile(publisherHandle, 0x08); // 0x01: Baseline, 0x02: Main, 0x08: High
// libPublisher.SetAVCHWEncoderLevel(publisherHandle, 0x200); // Level 3.1
// libPublisher.SetAVCHWEncoderLevel(publisherHandle, 0x400); // Level 3.2
// libPublisher.SetAVCHWEncoderLevel(publisherHandle, 0x800); // Level 4
libPublisher.SetAVCHWEncoderLevel(publisherHandle, 0x1000); // Level 4.1 多数情况下,这个够用了
//libPublisher.SetAVCHWEncoderLevel(publisherHandle, 0x2000); // Level 4.2
// libPublisher.SetVideoHWEncoderMaxBitrate(publisherHandle, ((long)h264HWKbps)*1300);
Log.i(TAG, "Great, it supports h.264 hardware encoder!");
}
}
else if (videoEncodeType == 2) {
int hevcHWKbps = setHardwareEncoderKbps(false, video_width_, video_height_);
hevcHWKbps = hevcHWKbps*fps/25;
Log.i(TAG, "hevcHWKbps: " + hevcHWKbps);
int isSupportHevcHWEncoder = libPublisher
.SetSmartPublisherVideoHevcHWEncoder(publisherHandle, hevcHWKbps);
if (isSupportHevcHWEncoder == 0) {
libPublisher.SetNativeMediaNDK(publisherHandle, 0);
libPublisher.SetVideoHWEncoderBitrateMode(publisherHandle, 0); // 0:CQ, 1:VBR, 2:CBR
libPublisher.SetVideoHWEncoderQuality(publisherHandle, 39);
// libPublisher.SetVideoHWEncoderMaxBitrate(publisherHandle, ((long)hevcHWKbps)*1200);
Log.i(TAG, "Great, it supports hevc hardware encoder!");
}
}
boolean is_sw_vbr_mode = true;
if(is_sw_vbr_mode) //H.264 software encoder
{
int is_enable_vbr = 1;
int video_quality = CalVideoQuality(video_width_, video_height_, true);
int vbr_max_bitrate = CalVbrMaxKBitRate(video_width_, video_height_);
libPublisher.SmartPublisherSetSwVBRMode(publisherHandle, is_enable_vbr, video_quality, vbr_max_bitrate);
}
if (is_pcma_) {
libPublisher.SmartPublisherSetAudioCodecType(publisherHandle, 3);
} else {
libPublisher.SmartPublisherSetAudioCodecType(publisherHandle, 1);
}
libPublisher.SetSmartPublisherEventCallbackV2(publisherHandle, new EventHandlerPublisherV2().set(handler_, recorder_io_executor_));
libPublisher.SmartPublisherSetSWVideoEncoderProfile(publisherHandle, 3);
libPublisher.SmartPublisherSetSWVideoEncoderSpeed(publisherHandle, 2);
libPublisher.SmartPublisherSetGopInterval(publisherHandle, gop);
libPublisher.SmartPublisherSetFPS(publisherHandle, fps);
// libPublisher.SmartPublisherSetSWVideoBitRate(publisherHandle, 600, 1200);
boolean is_noise_suppression = true;
libPublisher.SmartPublisherSetNoiseSuppression(publisherHandle, is_noise_suppression ? 1 : 0);
boolean is_agc = false;
libPublisher.SmartPublisherSetAGC(publisherHandle, is_agc ? 1 : 0);
int echo_cancel_delay = 0;
libPublisher.SmartPublisherSetEchoCancellation(publisherHandle, 1, echo_cancel_delay);
libPublisher.SmartPublisherSaveImageFlag(publisherHandle, 1);
}Dual Stream Encoding
Taking the capture of the camera as an example, if dual-stream encoding is required, when collecting the data source, the large resolution is used as the reference resolution for collection, such as 1920*1080, then if you need to upload real-time video data, you only need to zoom, Get encoded data with 1280*720 resolution:
@Override
public void onCameraImageData(Image image) {
if (null == libPublisher)
return;
if (isPushingRtmp || isRTSPPublisherRunning || isGB28181StreamRunning || isRecording) {
if (0 == publisherHandle)
return;
Image.Plane[] planes = image.getPlanes();
int w = image.getWidth(), h = image.getHeight();
int y_offset = 0, u_offset = 0, v_offset = 0;
Rect crop_rect = image.getCropRect();
if (crop_rect != null && !crop_rect.isEmpty()) {
w = crop_rect.width();
h = crop_rect.height();
y_offset += crop_rect.top * planes[0].getRowStride() + crop_rect.left * planes[0].getPixelStride();
u_offset += (crop_rect.top / 2) * planes[1].getRowStride() + (crop_rect.left / 2) * planes[1].getPixelStride();
v_offset += (crop_rect.top / 2) * planes[2].getRowStride() + (crop_rect.left / 2) * planes[2].getPixelStride();
;
// Log.i(TAG, "crop w:" + w + " h:" + h + " y_offset:"+ y_offset + " u_offset:" + u_offset + " v_offset:" + v_offset);
}
int scale_w = 0, scale_h = 0, scale_filter_mode = 0;
scale_filter_mode = 3;
int rotation_degree = cameraImageRotationDegree_;
if (rotation_degree < 0) {
Log.i(TAG, "onCameraImageData rotation_degree < 0, may need to set orientation_ to 0, 90, 180 or 270");
return;
}
if (!post_image_lock_.tryLock()) {
Log.i(TAG, "post_image_lock_.tryLock return false");
return;
}
try {
if (publisherHandle != 0) {
if (isPushingRtmp || isRTSPPublisherRunning || isGB28181StreamRunning || isRecording) {
libPublisher.PostLayerImageYUV420888ByteBuffer(publisherHandle, 0, 0, 0,
planes[0].getBuffer(), y_offset, planes[0].getRowStride(),
planes[1].getBuffer(), u_offset, planes[1].getRowStride(),
planes[2].getBuffer(), v_offset, planes[2].getRowStride(), planes[1].getPixelStride(),
w, h, 0, 0,
scale_w, scale_h, scale_filter_mode, rotation_degree);
}
}
}catch (Exception e) {
Log.e(TAG, "onCameraImageData Exception:", e);
}finally {
post_image_lock_.unlock();
}
}
}PostLayerImageYUV420888ByteBuffer() is designed as follows:
/*
* SmartPublisherJniV2.java
* SmartPublisherJniV2
*
* Author: https://daniusdk.com
* Created by DaniuLive on 2015/09/20.
*/
/**
* 投递层YUV420888图像, 专门为android.media.Image的android.graphics.ImageFormat.YUV_420_888格式提供的接口
*
* @param index: 层索引, 必须大于等于0
*
* @param left: 层叠加的左上角坐标, 对于第0层的话传0
*
* @param top: 层叠加的左上角坐标, 对于第0层的话传0
*
* @param y_plane: 对应android.media.Image.Plane[0].getBuffer()
*
* @param y_offset: 图像偏移, 这个主要目的是用来做clip的,一般传0
*
* @param y_row_stride: 对应android.media.Image.Plane[0].getRowStride()
*
* @param u_plane: android.media.Image.Plane[1].getBuffer()
*
* @param u_offset: 图像偏移, 这个主要目的是用来做clip的,一般传0
*
* @param u_row_stride: android.media.Image.Plane[1].getRowStride()
*
* @param v_plane: 对应android.media.Image.Plane[2].getBuffer()
*
* @param v_offset: 图像偏移, 这个主要目的是用来做clip的,一般传0
*
* @param v_row_stride: 对应android.media.Image.Plane[2].getRowStride()
*
* @param uv_pixel_stride: 对应android.media.Image.Plane[1].getPixelStride()
*
* @param width: width, 必须大于1, 且必须是偶数
*
* @param height: height, 必须大于1, 且必须是偶数
*
* @param is_vertical_flip: 是否垂直翻转, 0不翻转, 1翻转
*
* @param is_horizontal_flip：是否水平翻转, 0不翻转, 1翻转
*
* @param scale_width: 缩放宽，必须是偶数, 0或负数不缩放
*
* @param scale_height: 缩放高, 必须是偶数, 0或负数不缩放
*
* @param scale_filter_mode: 缩放质量, 传0使用默认速度,可选等级范围是:[1,3],值越大缩放质量越好, 但速度越慢
*
* @param rotation_degree: 顺时针旋转, 必须是0, 90, 180, 270, 注意:旋转是在缩放, 垂直/水品反转之后再做, 请留意顺序
*
* @return {0} if successful
*/
public native int PostLayerImageYUV420888ByteBuffer(long handle, int index, int left, int top,
ByteBuffer y_plane, int y_offset, int y_row_stride,
ByteBuffer u_plane, int u_offset, int u_row_stride,
ByteBuffer v_plane, int v_offset, int v_row_stride, int uv_pixel_stride,
int width, int height, int is_vertical_flip, int is_horizontal_flip,
int scale_width, int scale_height, int scale_filter_mode,
int rotation_degree);The above interface parameters, scale_width and scale_height can specify the zoom width and height, and even if the camera captures in the wrong direction, you can set the rotation_degree interface to realize the rotation of video data.
The first interface parameter is the instance handle. If two-way encoding is required, it must correspond to two push instances, that is, two handles, one for video recording and one for gb28181 upstream data push.
It should be noted that if two instances need to be encoded at the same time, when data needs to be delivered, the two instances call PostLayerImageYUV420888ByteBuffer() respectively to realize delivery from the data source to the underlying module.
The local recording operation is as follows:
class ButtonStartRecorderListener implements View.OnClickListener {
public void onClick(View v) {
if (isRecording) {
stopRecorder();
if (!isPushingRtmp && !isRTSPPublisherRunning && !isGB28181StreamRunning) {
ConfigControlEnable(true);
}
btnStartRecorder.setText("实时录像");
btnPauseRecorder.setText("暂停录像");
btnPauseRecorder.setEnabled(false);
isPauseRecording = true;
return;
}
Log.i(TAG, "onClick start recorder..");
if (libPublisher == null)
return;
if (!isPushingRtmp && !isRTSPPublisherRunning&& !isGB28181StreamRunning) {
InitAndSetConfig();
}
ConfigRecorderParam();
int startRet = libPublisher.SmartPublisherStartRecorder(publisherHandle);
if (startRet != 0) {
if (!isPushingRtmp && !isRTSPPublisherRunning && !isGB28181StreamRunning) {
if (publisherHandle != 0) {
long handle = publisherHandle;
publisherHandle = 0;
libPublisher.SmartPublisherClose(handle);
}
}
Log.e(TAG, "Failed to start recorder.");
return;
}
if (!isPushingRtmp && !isRTSPPublisherRunning && !isGB28181StreamRunning) {
CheckInitAudioRecorder();
ConfigControlEnable(false);
}
startLayerPostThread();
btnStartRecorder.setText("停止录像");
isRecording = true;
btnPauseRecorder.setEnabled(true);
isPauseRecording = true;
}
}Stop recording:
//停止录像
private void stopRecorder() {
if(!isRecording)
return;
isRecording = false;
if (!isPushingRtmp && !isRTSPPublisherRunning && !isGB28181StreamRunning)
stopLayerPostThread();
if (!isPushingRtmp && !isRTSPPublisherRunning && !isGB28181StreamRunning) {
if (audioRecord_ != null) {
Log.i(TAG, "stopRecorder, call audioRecord_.StopRecording..");
audioRecord_.Stop();
if (audioRecordCallback_ != null) {
audioRecord_.RemoveCallback(audioRecordCallback_);
audioRecordCallback_ = null;
}
audioRecord_ = null;
}
}
if (null == libPublisher || 0 == publisherHandle)
return;
libPublisher.SmartPublisherStopRecorder(publisherHandle);
if (!isPushingRtmp && !isRTSPPublisherRunning && !isGB28181StreamRunning) {
releasePublisherHandle();
}
}Technical Summary
On-demand encoding can be local video recording or upstream data push, corresponding to one instance. If dual-stream encoding is necessary, two instances are bound to be required, corresponding to different encoding parameters, and output H.264/H.265 data with different resolutions . In addition, where the audio data is called back, the two instances also call the audio delivery interface and pass it on. It should be noted that although the two-way video encoding can be hard-coded, it still puts higher requirements on the performance of the device.