我关注的播放器指标
好多开发者跟我交流音视频相关技术的时候,经常问我的问题是,多久可以开发个商业级别的RTMP或RTSP播放器?你们是怎样做到毫秒级延迟的?为什么一个播放器,会被你们做到那么复杂?带着这些疑问,结合Windows平台RTMP、RTSP播放模块,探讨下我的一点心得,不当之处权当抛砖引玉:
1. 低延迟:大多数RTMP或RTSP的播放都面向直播场景,如果延迟过大,严重影响体验,所以,低延迟是衡量一个好的直播播放器非常重要的指标,好多人对RTMP直播播放的印象,还停留在3-5秒的延迟,实际我们从2015年开发的RTMP播放来看,延迟也就几百毫秒,在一些强交互的场景,设置低延迟模式,甚至可以在200ms左右,RTSP延迟也在毫秒级,一些需要控制的场景,比如智能机器人、无人机等,实际使用下来,都可用满足场景诉求;
2. 音视频同步处理:大多播放器为了追求低延迟,甚至不做音视频同步,拿到audio video直接播放,导致音视频不同步,还有就是时间戳乱跳等各种问题,所以,一个好的直播播放器,需要有时间戳同步和异常时间戳矫正机制,当然,如果是超低延迟模式下,可以0 buffer,不做音视频同步:
3. 支持多实例:多实例RTMP、RTSP播放,是衡量一个直播播放器重要的指标,比如4-8-16路高分辨率播放;
4. 支持buffer time设置:在一些有网络抖动的场景,播放器需要支持buffer time设置;
5. 实时音量调节:比如,多窗口播放RTMP或RTSP流,如果每个audio都播放出来,体验非常不好,所以实时静音功能非常必要;
6. 视频view旋转:部分硬件设备,由于安装限制,导致图像倒置或旋转,所以一个好的RTMP、RTSP播放器应该支持如视频view实时旋转(0° 90° 180° 270°)、水平反转、垂直反转;
7. 支持解码后audio/video数据输出:好多开发者,希望能在播放的同时,获取到YUV或RGB数据,进行人脸匹配等算法分析,他们一个非常重要的诉求,就是高效率的获取到YUV或RGB数据;
8. 实时快照:感兴趣或重要的场景画面,实时截取下来非常必要;
9. 网络抖动处理(如断网重连):稳定的网络处理机制、支持如断网重连等,对直播播放器来说,非常重要;
10. 长期运行稳定性:7*24小时使用场景非常普遍,长时间运行稳定性的重要性不言而喻;
11. 实时下载速度反馈:提供音视频流实时下载回调,可设置回调时间间隔,确保实时下载速度反馈,以此来监听网络状态;
12. 异常状态处理、Event状态回调:如播放的过程中断网,我们提供的播放器可实时回调相关状态,确保上层模块感知处理,开源播放器对此支持不好;
13. 设置视频填充模式(等比例显示):好多情况下,有些场景需要全view铺满播放,有些为了防止视频拉伸,可以设置成等比例缩放显示;
14. D3D检测:一般来说市面上的大多Windows都支持D3D,有些小众化的,只支持GDI模式绘制,所以为了更好的兼容性,这个接口非常必要;
15. 特定机型硬解码:特定机型硬解码,也主要是用于多路播放场景下,通过硬解码,实现更低的CPU占用目的;
16. 只播放关键帧:特别是大屏多实例场景播放的时候,尽管我们已经CPU占用非常低了,如果只是查看大概的监控情景,实现更多路的播放,只播放关键帧是个非常好的功能点,如果需要原始帧播放,可以实时调节即可;
17. TCP-UDP设置:考虑到部分服务器或硬件设备或网络环境对TCP、UDP某一个支持的比较好,我们加了设置接口;
18. TCP-UDP自动切换:这个是更细力度的接口,比如默认设置了TCP模式,TCP模式下收不到数据,超时后,自动切换到UDP模式尝试,一般开源播放器不具备此功能;
19. RTSP超时时间设定:比如10-12秒收不到数据,自动重连,一般开源播放器支持不好。
技术实现
本文以大牛直播SDK的Windows平台C++的demo为例,探讨下RTMP、RTSP播放、录像、实时音量调节、快照等接口设计和处理:
模块初始化:
GetSmartPlayerSDKAPI(&player_api_);if ( NT_ERC_OK != player_api_.Init(0, NULL) ){return FALSE;}is_support_h264_hardware_decoder_ = NT_ERC_OK == player_api_.IsSupportH264HardwareDecoder();is_support_h265_hardware_decoder_ = NT_ERC_OK == player_api_.IsSupportH265HardwareDecoder();if ( NT_ERC_OK != player_api_.Open(&player_handle_, NULL, 0, NULL) ){return FALSE;}player_api_.SetEventCallBack(player_handle_, GetSafeHwnd(), &NT_SP_SDKEventHandle);
其他参数初始化:
bool CSmartPlayerDlg::InitCommonSDKParam()
{ASSERT(!is_playing_);ASSERT(!is_recording_);if ( NULL == player_handle_ )return false;CString wbuffer_str;edit_buffer.GetWindowTextW(wbuffer_str);std::wstring_convert<std::codecvt_utf8<wchar_t> > conv;auto buffer_str = conv.to_bytes(wbuffer_str);player_api_.SetBuffer(player_handle_, atoi(buffer_str.c_str()));// 设置rtsp 超时时间player_api_.SetRtspTimeout(player_handle_, rtsp_conf_info_.timeout_);// 设置rtsp tcp模式,rtmp不使用, 可以不设置player_api_.SetRTSPTcpMode(player_handle_, rtsp_conf_info_.is_tcp_ ? 1 : 0);player_api_.SetRtspAutoSwitchTcpUdp(player_handle_, rtsp_conf_info_.is_tcp_udp_auto_switch_ ? 1 : 0);if ( btn_check_fast_startup_.GetCheck() == BST_CHECKED ){player_api_.SetFastStartup(player_handle_, 1);}else{player_api_.SetFastStartup(player_handle_, 0);}player_api_.SetReportDownloadSpeed(player_handle_, 1, 1);if (NT_ERC_OK != player_api_.SetURL(player_handle_, GetURL().c_str())){return false;}connection_status_ = 0;buffer_status_ = 0;buffer_percent_ = 0;download_speed_ = -1;return true;
}
播放控制:
void CSmartPlayerDlg::OnBnClickedButtonPlay()
{if ( player_handle_ == NULL )return;CString btn_play_str;btn_play_.GetWindowTextW(btn_play_str);if ( btn_play_str == _T("播放") ){if ( !is_recording_ ){if ( !InitCommonSDKParam() ){AfxMessageBox(_T("设置参数错误!"));return;}}player_api_.SetVideoSizeCallBack(player_handle_, GetSafeHwnd(), SP_SDKVideoSizeHandle);bool is_support_d3d_render = false;NT_INT32 in_support_d3d_render = 0;if ( NT_ERC_OK == player_api_.IsSupportD3DRender(player_handle_,wrapper_render_wnd_.RenderWnd(), &in_support_d3d_render)){if ( 1 == in_support_d3d_render ){is_support_d3d_render = true;}}if ( is_support_d3d_render ){is_gdi_render_ = false;// 支持d3d绘制的话,就用D3D绘制player_api_.SetRenderWindow(player_handle_, wrapper_render_wnd_.RenderWnd());player_api_.SetRenderScaleMode(player_handle_, btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);}else{is_gdi_render_ = true;// 不支持D3D就让播放器吐出数据来,用GDI绘制wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);player_api_.SetVideoFrameCallBack(player_handle_, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32,GetSafeHwnd(), SM_SDKVideoFrameHandle);}if ( BST_CHECKED == btn_check_hardware_decoder_.GetCheck() ){player_api_.SetH264HardwareDecoder(player_handle_, is_support_h264_hardware_decoder_?1:0, 0);player_api_.SetH265HardwareDecoder(player_handle_, is_support_h265_hardware_decoder_?1:0, 0);}else{player_api_.SetH264HardwareDecoder(player_handle_, 0, 0);player_api_.SetH265HardwareDecoder(player_handle_, 0, 0);}player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);player_api_.SetLowLatencyMode(player_handle_, BST_CHECKED == btn_check_low_latency_.GetCheck() ? 1 : 0);player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 :0 );player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);player_api_.SetRotation(player_handle_, rotate_degrees_);player_api_.SetAudioVolume(player_handle_, slider_audio_volume_.GetPos());player_api_.SetUserDataCallBack(player_handle_, GetSafeHwnd(), NT_SP_SDKUserDataHandle);if (NT_ERC_OK != player_api_.StartPlay(player_handle_)){AfxMessageBox(_T("播放器失败!"));return;}btn_play_.SetWindowTextW(_T("停止"));is_playing_ = true;}else{StopPlayback();}
}
实时快照:
void CSmartPlayerDlg::OnBnClickedButtonCaptureImage()
{if ( capture_image_path_.empty() ){AfxMessageBox(_T("请先设置保存截图文件的目录! 点击截图左边的按钮设置!"));return;}if ( player_handle_ == NULL ){return;}if ( !is_playing_ ){return;}std::wostringstream ss;ss << capture_image_path_;if ( capture_image_path_.back() != L'\\' ){ss << L"\\";}SYSTEMTIME sysTime;::GetLocalTime(&sysTime);ss << L"SmartPlayer-"<< std::setfill(L'0') << std::setw(4) << sysTime.wYear<< std::setfill(L'0') << std::setw(2) << sysTime.wMonth<< std::setfill(L'0') << std::setw(2) << sysTime.wDay<< L"-"<< std::setfill(L'0') << std::setw(2) << sysTime.wHour<< std::setfill(L'0') << std::setw(2) << sysTime.wMinute<< std::setfill(L'0') << std::setw(2) << sysTime.wSecond;ss << L"-" << std::setfill(L'0') << std::setw(3) << sysTime.wMilliseconds<< L".png";std::wstring_convert<std::codecvt_utf8<wchar_t> > conv;auto val_str = conv.to_bytes(ss.str());auto ret = player_api_.CaptureImage(player_handle_, val_str.c_str(), NULL, &SM_SDKCaptureImageHandle);if (NT_ERC_OK == ret){// 发送截图请求成功}else if (NT_ERC_SP_TOO_MANY_CAPTURE_IMAGE_REQUESTS == ret){// 通知用户延时OutputDebugStringA("Too many capture image requests!!!\r\n");}else{// 其他失败}
}
只解关键帧:
void CSmartPlayerDlg::OnBnClickedCheckOnlyDecodeVideoKeyFrame()
{if (player_handle_ != NULL){player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);}
}
设置等比例显示或全铺满模式:
void CSmartPlayerDlg::OnBnClickedCheckRenderScaleMode()
{if (player_handle_ != NULL){if (!is_gdi_render_){player_api_.SetRenderScaleMode(player_handle_, BST_CHECKED == btn_check_render_scale_mode_.GetCheck() ? 1 : 0);}else{wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);}}
}
视频view水平反转、垂直反转、旋转:
void CSmartPlayerDlg::OnBnClickedCheckFlipHorizontal()
{if (player_handle_ != NULL){player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);}
}void CSmartPlayerDlg::OnBnClickedCheckFlipVertical()
{if (player_handle_ != NULL){player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 : 0);}
}void CSmartPlayerDlg::OnBnClickedButtonRotation()
{rotate_degrees_ += 90;rotate_degrees_ = rotate_degrees_ % 360;if (0 == rotate_degrees_){btn_rotation_.SetWindowText(_T("旋转90度"));}else if (90 == rotate_degrees_){btn_rotation_.SetWindowText(_T("旋转180度"));}else if (180 == rotate_degrees_){btn_rotation_.SetWindowText(_T("旋转270度"));}else if (270 == rotate_degrees_){btn_rotation_.SetWindowText(_T("不旋转"));}if ( player_handle_ != NULL ){player_api_.SetRotation(player_handle_, rotate_degrees_);}
}
实时录像:
void CSmartPlayerDlg::OnBnClickedButtonRecord()
{if ( player_handle_ == NULL )return;CString btn_record_str;btn_record_.GetWindowTextW(btn_record_str);if ( btn_record_str == _T("录像") ){if ( !rec_conf_info_.is_record_video_ && !rec_conf_info_.is_record_audio_ ){AfxMessageBox(_T("音频录制选项和视频录制选项至少需要选择一个!"));return;}if ( !is_playing_ ){if ( !InitCommonSDKParam() ){AfxMessageBox(_T("设置参数错误!"));return;}}player_api_.SetRecorderVideo(player_handle_, rec_conf_info_.is_record_video_ ? 1 : 0);player_api_.SetRecorderAudio(player_handle_, rec_conf_info_.is_record_audio_ ? 1 : 0);auto ret = player_api_.SetRecorderDirectory(player_handle_, rec_conf_info_.dir_.c_str());if ( NT_ERC_OK != ret ){AfxMessageBox(_T("设置录像目录失败,请确保目录存在且是英文目录"));return;}player_api_.SetRecorderFileMaxSize(player_handle_, rec_conf_info_.file_max_size_);NT_SP_RecorderFileNameRuler rec_name_ruler = { 0 };rec_name_ruler.type_ = 0;rec_name_ruler.file_name_prefix_ = rec_conf_info_.file_name_prefix_.c_str();rec_name_ruler.append_date_ = rec_conf_info_.is_append_date_ ? 1 : 0;rec_name_ruler.append_time_ = rec_conf_info_.is_append_time_ ? 1 : 0;player_api_.SetRecorderFileNameRuler(player_handle_, &rec_name_ruler);player_api_.SetRecorderCallBack(player_handle_, GetSafeHwnd(), &SP_SDKRecorderHandle);player_api_.SetRecorderAudioTranscodeAAC(player_handle_, rec_conf_info_.is_audio_transcode_aac_ ? 1 : 0);if ( NT_ERC_OK != player_api_.StartRecorder(player_handle_) ){AfxMessageBox(_T("录像失败!"));return;}btn_record_.SetWindowTextW(_T("停止录像"));is_recording_ = true;}else{StopRecorder();}
}
Event回调:
LRESULT CSmartPlayerDlg::OnSDKEvent(WPARAM wParam, LPARAM lParam)
{if (!is_playing_ && !is_recording_){return S_OK;}NT_UINT32 event_id = (NT_UINT32)(wParam);if ( NT_SP_E_EVENT_ID_PLAYBACK_REACH_EOS == event_id ){StopPlayback();return S_OK;}else if ( NT_SP_E_EVENT_ID_RECORDER_REACH_EOS == event_id ){StopRecorder();return S_OK;}else if ( NT_SP_E_EVENT_ID_RTSP_STATUS_CODE == event_id ){int status_code = (int)lParam;if ( 401 == status_code ){HandleVerification();}return S_OK;}else if (NT_SP_E_EVENT_ID_NEED_KEY == event_id){HandleKeyEvent(false);return S_OK;}else if (NT_SP_E_EVENT_ID_KEY_ERROR == event_id){HandleKeyEvent(true);return S_OK;}else if ( NT_SP_E_EVENT_ID_PULLSTREAM_REACH_EOS == event_id ){if (player_handle_ != NULL){player_api_.StopPullStream(player_handle_);}return S_OK;}else if ( NT_SP_E_EVENT_ID_DURATION == event_id ){NT_INT64 duration = (NT_INT64)(lParam);edit_duration_.SetWindowTextW(GetHMSMsFormatStr(duration, false, false).c_str());return S_OK;}if ( NT_SP_E_EVENT_ID_CONNECTING == event_id|| NT_SP_E_EVENT_ID_CONNECTION_FAILED == event_id|| NT_SP_E_EVENT_ID_CONNECTED == event_id|| NT_SP_E_EVENT_ID_DISCONNECTED == event_id|| NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == event_id){if ( NT_SP_E_EVENT_ID_CONNECTING == event_id ){OutputDebugStringA("connection status: connecting\r\n");}else if ( NT_SP_E_EVENT_ID_CONNECTION_FAILED == event_id ){OutputDebugStringA("connection status: connection failed\r\n");}else if ( NT_SP_E_EVENT_ID_CONNECTED == event_id ){OutputDebugStringA("connection status: connected\r\n");}else if (NT_SP_E_EVENT_ID_DISCONNECTED == event_id){OutputDebugStringA("connection status: disconnected\r\n");}else if (NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == event_id){OutputDebugStringA("connection status: no mediadata received\r\n");}connection_status_ = event_id;}if ( NT_SP_E_EVENT_ID_START_BUFFERING == event_id|| NT_SP_E_EVENT_ID_BUFFERING == event_id|| NT_SP_E_EVENT_ID_STOP_BUFFERING == event_id ){buffer_status_ = event_id;if ( NT_SP_E_EVENT_ID_BUFFERING == event_id ){buffer_percent_ = (NT_INT32)lParam;std::wostringstream ss;ss << L"buffering:" << buffer_percent_ << "%";OutputDebugStringW(ss.str().c_str());OutputDebugStringW(L"\r\n");}}if ( NT_SP_E_EVENT_ID_DOWNLOAD_SPEED == event_id ){download_speed_ = (NT_INT32)lParam;}CString show_str = base_title_;if ( connection_status_ != 0 ){show_str += _T("--链接状态: ");if ( NT_SP_E_EVENT_ID_CONNECTING == connection_status_ ){show_str += _T("链接中");}else if ( NT_SP_E_EVENT_ID_CONNECTION_FAILED == connection_status_ ){show_str += _T("链接失败");}else if ( NT_SP_E_EVENT_ID_CONNECTED == connection_status_ ){show_str += _T("链接成功");}else if ( NT_SP_E_EVENT_ID_DISCONNECTED == connection_status_ ){show_str += _T("链接断开");}else if (NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == connection_status_){show_str += _T("收不到数据");}}if (download_speed_ != -1){std::wostringstream ss;ss << L"--下载速度:" << (download_speed_ * 8 / 1000) << "kbps"<< L"(" << (download_speed_ / 1024) << "KB/s)";show_str += ss.str().c_str();}if ( buffer_status_ != 0 ){show_str += _T("--缓冲状态: ");if ( NT_SP_E_EVENT_ID_START_BUFFERING == buffer_status_ ){show_str += _T("开始缓冲");}else if (NT_SP_E_EVENT_ID_BUFFERING == buffer_status_){std::wostringstream ss;ss << L"缓冲中" << buffer_percent_ << "%";show_str += ss.str().c_str();}else if (NT_SP_E_EVENT_ID_STOP_BUFFERING == buffer_status_){show_str += _T("结束缓冲");}}SetWindowText(show_str);return S_OK;
}
总结
我们常说,做RTMP或RTSP播放器容易,但是做个高稳定、低延迟、功能强大的直播播放器,特别是细节处理,还是有一定的难度,开源播放器在点播播放问题不大,直播场景下,欠缺还很多,做播放器容易,做好播放器,难。上述只是一点经验分享,感兴趣的开发者,可以参考。