Player metrics I follow
When many developers communicate with me about audio and video related technologies, the question they often ask me is, how long will it take to develop a commercial-level RTMP or RTSP player? How do you achieve millisecond latency? Why is a player so complicated? With these questions in mind, combined with the RTMP and RTSP playback modules of the Windows platform, I will discuss some of my experiences and point out any inappropriate points:
1. Low latency: Most RTMP or RTSP playback is for live broadcast scenarios. If the delay is too large, it will seriously affect the experience. Therefore, low latency is a very important indicator for measuring a good live broadcast player. Many people are interested in RTMP live broadcast. The impression is still stuck at a delay of 3-5 seconds. In fact, judging from the RTMP playback developed in 2015, the delay is only a few hundred milliseconds. In some strongly interactive scenes, setting a low-latency mode can even be around 200ms, RTSP The delay is also at the millisecond level. Some scenes that need to be controlled, such as intelligent robots, drones, etc., can be used to meet the scene demands after actual use;
2. Audio and video synchronization processing: In order to pursue low latency, most players do not even synchronize audio and video. They get the audio video and play it directly, resulting in audio and video out of synchronization, and various problems such as random jumping of timestamps. Therefore, a A good live broadcast player needs to have timestamp synchronization and abnormal timestamp correction mechanisms. Of course, if it is in ultra-low latency mode, it can have 0 buffer and no audio and video synchronization:
3. Support multiple instances: multi-instance RTMP and RTSP playback are important indicators for measuring a live broadcast player, such as 4-8-16 high-resolution playback;
4. Support buffer time setting: In some scenarios with network jitter, the player needs to support buffer time setting;
5. Real-time volume adjustment: For example, when playing RTMP or RTSP streams in multiple windows, if every audio is played, the experience will be very bad, so the real-time mute function is very necessary;
6. Video view rotation: Some hardware devices, due to installation restrictions, cause the image to be inverted or rotated. Therefore, a good RTMP and RTSP player should support real-time rotation of the video view (0° 90° 180° 270°) and horizontal inversion. , vertical inversion;
7. Support decoded audio/video data output: Many developers hope to obtain YUV or RGB data while playing, and perform algorithm analysis such as face matching. One of their very important demands is to obtain the data efficiently. YUV or RGB data;
8. Real-time snapshots: It is very necessary to capture interesting or important scenes in real time;
9. Network jitter processing (such as network disconnection and reconnection): Stable network processing mechanism and support such as network disconnection and reconnection are very important for live broadcast players;
10. Long-term operation stability: 7*24-hour usage scenarios are very common, and the importance of long-term operation stability is self-evident;
11. Real-time download speed feedback: Provides real-time download callback for audio and video streams. The callback time interval can be set to ensure real-time download speed feedback to monitor network status;
12. Exception status processing, Event status callback: If the network is interrupted during playback, the player we provide can call back the relevant status in real time to ensure that the upper module is aware of the processing. Open source players do not support this well;
13. Set the video filling mode (display in equal proportions): In many cases, some scenes require the entire view to be played. In some cases, in order to prevent the video from stretching, it can be set to display in equal proportions;
14. D3D detection: Generally speaking, most Windows on the market support D3D, and some niche ones only support GDI mode drawing, so for better compatibility, this interface is very necessary;
15. Hard decoding of specific models: Hard decoding of specific models is mainly used in multi-channel playback scenarios to achieve lower CPU usage through hard decoding;
16. Only play key frames: Especially when playing large-screen multi-instance scenes, although our CPU usage is very low, if you only want to view the approximate monitoring situation and achieve more channels of playback, it is very good to only play key frames. Function point, if you need to play the original frame, you can adjust it in real time;
17. TCP-UDP settings: Considering that some servers or hardware devices or network environments support TCP or UDP better, we have added a setting interface;
18. TCP-UDP automatic switching: This is a more detailed interface. For example, the TCP mode is set by default. Data cannot be received in TCP mode. After a timeout, it will automatically switch to UDP mode to try. Generally, open source players do not have this function;
19. RTSP timeout setting: For example, if data cannot be received within 10-12 seconds, it will automatically reconnect. Generally, open source players do not support it well.
Technical realization
This article takes the Windows platform C++ demo of Daniu Live SDK as an example to discuss the design and processing of interfaces such as RTMP, RTSP playback, recording, real-time volume adjustment, and snapshots:
Module initialization:
GetSmartPlayerSDKAPI(&player_api_);
if ( NT_ERC_OK != player_api_.Init(0, NULL) )
{
return FALSE;
}
is_support_h264_hardware_decoder_ = NT_ERC_OK == player_api_.IsSupportH264HardwareDecoder();
is_support_h265_hardware_decoder_ = NT_ERC_OK == player_api_.IsSupportH265HardwareDecoder();
if ( NT_ERC_OK != player_api_.Open(&player_handle_, NULL, 0, NULL) )
{
return FALSE;
}
player_api_.SetEventCallBack(player_handle_, GetSafeHwnd(), &NT_SP_SDKEventHandle);Other parameter initialization:
bool CSmartPlayerDlg::InitCommonSDKParam()
{
ASSERT(!is_playing_);
ASSERT(!is_recording_);
if ( NULL == player_handle_ )
return false;
CString wbuffer_str;
edit_buffer.GetWindowTextW(wbuffer_str);
std::wstring_convert<std::codecvt_utf8<wchar_t> > conv;
auto buffer_str = conv.to_bytes(wbuffer_str);
player_api_.SetBuffer(player_handle_, atoi(buffer_str.c_str()));
// 设置rtsp 超时时间
player_api_.SetRtspTimeout(player_handle_, rtsp_conf_info_.timeout_);
// 设置rtsp tcp模式，rtmp不使用, 可以不设置
player_api_.SetRTSPTcpMode(player_handle_, rtsp_conf_info_.is_tcp_ ? 1 : 0);
player_api_.SetRtspAutoSwitchTcpUdp(player_handle_, rtsp_conf_info_.is_tcp_udp_auto_switch_ ? 1 : 0);
if ( btn_check_fast_startup_.GetCheck() == BST_CHECKED )
{
player_api_.SetFastStartup(player_handle_, 1);
}
else
{
player_api_.SetFastStartup(player_handle_, 0);
}
player_api_.SetReportDownloadSpeed(player_handle_, 1, 1);
if (NT_ERC_OK != player_api_.SetURL(player_handle_, GetURL().c_str()))
{
return false;
}
connection_status_ = 0;
buffer_status_ = 0;
buffer_percent_ = 0;
download_speed_ = -1;
return true;
}Playback controls:
void CSmartPlayerDlg::OnBnClickedButtonPlay()
{
if ( player_handle_ == NULL )
return;
CString btn_play_str;
btn_play_.GetWindowTextW(btn_play_str);
if ( btn_play_str == _T("播放") )
{
if ( !is_recording_ )
{
if ( !InitCommonSDKParam() )
{
AfxMessageBox(_T("设置参数错误!"));
return;
}
}
player_api_.SetVideoSizeCallBack(player_handle_, GetSafeHwnd(), SP_SDKVideoSizeHandle);
bool is_support_d3d_render = false;
NT_INT32 in_support_d3d_render = 0;
if ( NT_ERC_OK == player_api_.IsSupportD3DRender(player_handle_,
wrapper_render_wnd_.RenderWnd(), &in_support_d3d_render))
{
if ( 1 == in_support_d3d_render )
{
is_support_d3d_render = true;
}
}
if ( is_support_d3d_render )
{
is_gdi_render_ = false;
// 支持d3d绘制的话，就用D3D绘制
player_api_.SetRenderWindow(player_handle_, wrapper_render_wnd_.RenderWnd());
player_api_.SetRenderScaleMode(player_handle_, btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);
}
else
{
is_gdi_render_ = true;
// 不支持D3D就让播放器吐出数据来，用GDI绘制
wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);
player_api_.SetVideoFrameCallBack(player_handle_, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32,
GetSafeHwnd(), SM_SDKVideoFrameHandle);
}
if ( BST_CHECKED == btn_check_hardware_decoder_.GetCheck() )
{
player_api_.SetH264HardwareDecoder(player_handle_, is_support_h264_hardware_decoder_?1:0, 0);
player_api_.SetH265HardwareDecoder(player_handle_, is_support_h265_hardware_decoder_?1:0, 0);
}
else
{
player_api_.SetH264HardwareDecoder(player_handle_, 0, 0);
player_api_.SetH265HardwareDecoder(player_handle_, 0, 0);
}
player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);
player_api_.SetLowLatencyMode(player_handle_, BST_CHECKED == btn_check_low_latency_.GetCheck() ? 1 : 0);
player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 :0 );
player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);
player_api_.SetRotation(player_handle_, rotate_degrees_);
player_api_.SetAudioVolume(player_handle_, slider_audio_volume_.GetPos());
player_api_.SetUserDataCallBack(player_handle_, GetSafeHwnd(), NT_SP_SDKUserDataHandle);
if (NT_ERC_OK != player_api_.StartPlay(player_handle_))
{
AfxMessageBox(_T("播放器失败!"));
return;
}
btn_play_.SetWindowTextW(_T("停止"));
is_playing_ = true;
}
else
{
StopPlayback();
}
}Live snapshot:
void CSmartPlayerDlg::OnBnClickedButtonCaptureImage()
{
if ( capture_image_path_.empty() )
{
AfxMessageBox(_T("请先设置保存截图文件的目录! 点击截图左边的按钮设置!"));
return;
}
if ( player_handle_ == NULL )
{
return;
}
if ( !is_playing_ )
{
return;
}
std::wostringstream ss;
ss << capture_image_path_;
if ( capture_image_path_.back() != L'\\' )
{
ss << L"\\";
}
SYSTEMTIME sysTime;
::GetLocalTime(&sysTime);
ss << L"SmartPlayer-"
<< std::setfill(L'0') << std::setw(4) << sysTime.wYear
<< std::setfill(L'0') << std::setw(2) << sysTime.wMonth
<< std::setfill(L'0') << std::setw(2) << sysTime.wDay
<< L"-"
<< std::setfill(L'0') << std::setw(2) << sysTime.wHour
<< std::setfill(L'0') << std::setw(2) << sysTime.wMinute
<< std::setfill(L'0') << std::setw(2) << sysTime.wSecond;
ss << L"-" << std::setfill(L'0') << std::setw(3) << sysTime.wMilliseconds
<< L".png";
std::wstring_convert<std::codecvt_utf8<wchar_t> > conv;
auto val_str = conv.to_bytes(ss.str());
auto ret = player_api_.CaptureImage(player_handle_, val_str.c_str(), NULL, &SM_SDKCaptureImageHandle);
if (NT_ERC_OK == ret)
{
// 发送截图请求成功
}
else if (NT_ERC_SP_TOO_MANY_CAPTURE_IMAGE_REQUESTS == ret)
{
// 通知用户延时
OutputDebugStringA("Too many capture image requests!!!\r\n");
}
else
{
// 其他失败
}
}Only solve keyframes:
void CSmartPlayerDlg::OnBnClickedCheckOnlyDecodeVideoKeyFrame()
{
if (player_handle_ != NULL)
{
player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);
}
}Set proportional display or full tile mode:
void CSmartPlayerDlg::OnBnClickedCheckRenderScaleMode()
{
if (player_handle_ != NULL)
{
if (!is_gdi_render_)
{
player_api_.SetRenderScaleMode(player_handle_, BST_CHECKED == btn_check_render_scale_mode_.GetCheck() ? 1 : 0);
}
else
{
wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);
}
}
}Video view horizontal inversion, vertical inversion, rotation:
void CSmartPlayerDlg::OnBnClickedCheckFlipHorizontal()
{
if (player_handle_ != NULL)
{
player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);
}
}
void CSmartPlayerDlg::OnBnClickedCheckFlipVertical()
{
if (player_handle_ != NULL)
{
player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 : 0);
}
}
void CSmartPlayerDlg::OnBnClickedButtonRotation()
{
rotate_degrees_ += 90;
rotate_degrees_ = rotate_degrees_ % 360;
if (0 == rotate_degrees_)
{
btn_rotation_.SetWindowText(_T("旋转90度"));
}
else if (90 == rotate_degrees_)
{
btn_rotation_.SetWindowText(_T("旋转180度"));
}
else if (180 == rotate_degrees_)
{
btn_rotation_.SetWindowText(_T("旋转270度"));
}
else if (270 == rotate_degrees_)
{
btn_rotation_.SetWindowText(_T("不旋转"));
}
if ( player_handle_ != NULL )
{
player_api_.SetRotation(player_handle_, rotate_degrees_);
}
}Real-time recording:
void CSmartPlayerDlg::OnBnClickedButtonRecord()
{
if ( player_handle_ == NULL )
return;
CString btn_record_str;
btn_record_.GetWindowTextW(btn_record_str);
if ( btn_record_str == _T("录像") )
{
if ( !rec_conf_info_.is_record_video_ && !rec_conf_info_.is_record_audio_ )
{
AfxMessageBox(_T("音频录制选项和视频录制选项至少需要选择一个!"));
return;
}
if ( !is_playing_ )
{
if ( !InitCommonSDKParam() )
{
AfxMessageBox(_T("设置参数错误!"));
return;
}
}
player_api_.SetRecorderVideo(player_handle_, rec_conf_info_.is_record_video_ ? 1 : 0);
player_api_.SetRecorderAudio(player_handle_, rec_conf_info_.is_record_audio_ ? 1 : 0);
auto ret = player_api_.SetRecorderDirectory(player_handle_, rec_conf_info_.dir_.c_str());
if ( NT_ERC_OK != ret )
{
AfxMessageBox(_T("设置录像目录失败，请确保目录存在且是英文目录"));
return;
}
player_api_.SetRecorderFileMaxSize(player_handle_, rec_conf_info_.file_max_size_);
NT_SP_RecorderFileNameRuler rec_name_ruler = { 0 };
rec_name_ruler.type_ = 0;
rec_name_ruler.file_name_prefix_ = rec_conf_info_.file_name_prefix_.c_str();
rec_name_ruler.append_date_ = rec_conf_info_.is_append_date_ ? 1 : 0;
rec_name_ruler.append_time_ = rec_conf_info_.is_append_time_ ? 1 : 0;
player_api_.SetRecorderFileNameRuler(player_handle_, &rec_name_ruler);
player_api_.SetRecorderCallBack(player_handle_, GetSafeHwnd(), &SP_SDKRecorderHandle);
player_api_.SetRecorderAudioTranscodeAAC(player_handle_, rec_conf_info_.is_audio_transcode_aac_ ? 1 : 0);
if ( NT_ERC_OK != player_api_.StartRecorder(player_handle_) )
{
AfxMessageBox(_T("录像失败!"));
return;
}
btn_record_.SetWindowTextW(_T("停止录像"));
is_recording_ = true;
}
else
{
StopRecorder();
}
}Event callback:
LRESULT CSmartPlayerDlg::OnSDKEvent(WPARAM wParam, LPARAM lParam)
{
if (!is_playing_ && !is_recording_)
{
return S_OK;
}
NT_UINT32 event_id = (NT_UINT32)(wParam);
if ( NT_SP_E_EVENT_ID_PLAYBACK_REACH_EOS == event_id )
{
StopPlayback();
return S_OK;
}
else if ( NT_SP_E_EVENT_ID_RECORDER_REACH_EOS == event_id )
{
StopRecorder();
return S_OK;
}
else if ( NT_SP_E_EVENT_ID_RTSP_STATUS_CODE == event_id )
{
int status_code = (int)lParam;
if ( 401 == status_code )
{
HandleVerification();
}
return S_OK;
}
else if (NT_SP_E_EVENT_ID_NEED_KEY == event_id)
{
HandleKeyEvent(false);
return S_OK;
}
else if (NT_SP_E_EVENT_ID_KEY_ERROR == event_id)
{
HandleKeyEvent(true);
return S_OK;
}
else if ( NT_SP_E_EVENT_ID_PULLSTREAM_REACH_EOS == event_id )
{
if (player_handle_ != NULL)
{
player_api_.StopPullStream(player_handle_);
}
return S_OK;
}
else if ( NT_SP_E_EVENT_ID_DURATION == event_id )
{
NT_INT64 duration = (NT_INT64)(lParam);
edit_duration_.SetWindowTextW(GetHMSMsFormatStr(duration, false, false).c_str());
return S_OK;
}
if ( NT_SP_E_EVENT_ID_CONNECTING == event_id
|| NT_SP_E_EVENT_ID_CONNECTION_FAILED == event_id
|| NT_SP_E_EVENT_ID_CONNECTED == event_id
|| NT_SP_E_EVENT_ID_DISCONNECTED == event_id
|| NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == event_id)
{
if ( NT_SP_E_EVENT_ID_CONNECTING == event_id )
{
OutputDebugStringA("connection status: connecting\r\n");
}
else if ( NT_SP_E_EVENT_ID_CONNECTION_FAILED == event_id )
{
OutputDebugStringA("connection status: connection failed\r\n");
}
else if ( NT_SP_E_EVENT_ID_CONNECTED == event_id )
{
OutputDebugStringA("connection status: connected\r\n");
}
else if (NT_SP_E_EVENT_ID_DISCONNECTED == event_id)
{
OutputDebugStringA("connection status: disconnected\r\n");
}
else if (NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == event_id)
{
OutputDebugStringA("connection status: no mediadata received\r\n");
}
connection_status_ = event_id;
}
if ( NT_SP_E_EVENT_ID_START_BUFFERING == event_id
|| NT_SP_E_EVENT_ID_BUFFERING == event_id
|| NT_SP_E_EVENT_ID_STOP_BUFFERING == event_id )
{
buffer_status_ = event_id;
if ( NT_SP_E_EVENT_ID_BUFFERING == event_id )
{
buffer_percent_ = (NT_INT32)lParam;
std::wostringstream ss;
ss << L"buffering:" << buffer_percent_ << "%";
OutputDebugStringW(ss.str().c_str());
OutputDebugStringW(L"\r\n");
}
}
if ( NT_SP_E_EVENT_ID_DOWNLOAD_SPEED == event_id )
{
download_speed_ = (NT_INT32)lParam;
}
CString show_str = base_title_;
if ( connection_status_ != 0 )
{
show_str += _T("--链接状态: ");
if ( NT_SP_E_EVENT_ID_CONNECTING == connection_status_ )
{
show_str += _T("链接中");
}
else if ( NT_SP_E_EVENT_ID_CONNECTION_FAILED == connection_status_ )
{
show_str += _T("链接失败");
}
else if ( NT_SP_E_EVENT_ID_CONNECTED == connection_status_ )
{
show_str += _T("链接成功");
}
else if ( NT_SP_E_EVENT_ID_DISCONNECTED == connection_status_ )
{
show_str += _T("链接断开");
}
else if (NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == connection_status_)
{
show_str += _T("收不到数据");
}
}
if (download_speed_ != -1)
{
std::wostringstream ss;
ss << L"--下载速度:" << (download_speed_ * 8 / 1000) << "kbps"
<< L"(" << (download_speed_ / 1024) << "KB/s)";
show_str += ss.str().c_str();
}
if ( buffer_status_ != 0 )
{
show_str += _T("--缓冲状态: ");
if ( NT_SP_E_EVENT_ID_START_BUFFERING == buffer_status_ )
{
show_str += _T("开始缓冲");
}
else if (NT_SP_E_EVENT_ID_BUFFERING == buffer_status_)
{
std::wostringstream ss;
ss << L"缓冲中" << buffer_percent_ << "%";
show_str += ss.str().c_str();
}
else if (NT_SP_E_EVENT_ID_STOP_BUFFERING == buffer_status_)
{
show_str += _T("结束缓冲");
}
}
SetWindowText(show_str);
return S_OK;
}Summarize
We often say that it is easy to make an RTMP or RTSP player, but it is still difficult to make a high-stable, low-latency, and powerful live broadcast player, especially in detail processing. Open source players do not have big problems in on-demand playback, and live broadcasts In this scenario, there are still many shortcomings. It is easy to make a player, but it is difficult to make a good player. The above is just a little experience sharing, and interested developers can refer to it.