对话音视频牛哥：开发RTSP|RTMP直播播放器难不难？难在哪？

我关注的播放器指标

好多开发者跟我交流音视频相关技术的时候，经常问我的问题是，多久可以开发个商业级别的RTMP或RTSP播放器？你们是怎样做到毫秒级延迟的？为什么一个播放器，会被你们做到那么复杂？带着这些疑问，结合Windows平台RTMP、RTSP播放模块，探讨下我的一点心得，不当之处权当抛砖引玉：

1. 低延迟：大多数RTMP或RTSP的播放都面向直播场景，如果延迟过大，严重影响体验，所以，低延迟是衡量一个好的直播播放器非常重要的指标，好多人对RTMP直播播放的印象，还停留在3-5秒的延迟，实际我们从2015年开发的RTMP播放来看，延迟也就几百毫秒，在一些强交互的场景，设置低延迟模式，甚至可以在200ms左右，RTSP延迟也在毫秒级，一些需要控制的场景，比如智能机器人、无人机等，实际使用下来，都可用满足场景诉求；

2. 音视频同步处理：大多播放器为了追求低延迟，甚至不做音视频同步，拿到audio video直接播放，导致音视频不同步，还有就是时间戳乱跳等各种问题，所以，一个好的直播播放器，需要有时间戳同步和异常时间戳矫正机制，当然，如果是超低延迟模式下，可以0 buffer，不做音视频同步：

3. 支持多实例：多实例RTMP、RTSP播放，是衡量一个直播播放器重要的指标，比如4-8-16路高分辨率播放；

4. 支持buffer time设置：在一些有网络抖动的场景，播放器需要支持buffer time设置；

5. 实时音量调节：比如，多窗口播放RTMP或RTSP流，如果每个audio都播放出来，体验非常不好，所以实时静音功能非常必要；

6. 视频view旋转：部分硬件设备，由于安装限制，导致图像倒置或旋转，所以一个好的RTMP、RTSP播放器应该支持如视频view实时旋转(0° 90° 180° 270°)、水平反转、垂直反转；

7. 支持解码后audio/video数据输出：好多开发者，希望能在播放的同时，获取到YUV或RGB数据，进行人脸匹配等算法分析，他们一个非常重要的诉求，就是高效率的获取到YUV或RGB数据；

8. 实时快照：感兴趣或重要的场景画面，实时截取下来非常必要；

9. 网络抖动处理(如断网重连)：稳定的网络处理机制、支持如断网重连等，对直播播放器来说，非常重要；

10. 长期运行稳定性：7*24小时使用场景非常普遍，长时间运行稳定性的重要性不言而喻；

11. 实时下载速度反馈：提供音视频流实时下载回调，可设置回调时间间隔，确保实时下载速度反馈，以此来监听网络状态；

12. 异常状态处理、Event状态回调：如播放的过程中断网，我们提供的播放器可实时回调相关状态，确保上层模块感知处理，开源播放器对此支持不好；

13. 设置视频填充模式（等比例显示）：好多情况下，有些场景需要全view铺满播放，有些为了防止视频拉伸，可以设置成等比例缩放显示；

14. D3D检测：一般来说市面上的大多Windows都支持D3D，有些小众化的，只支持GDI模式绘制，所以为了更好的兼容性，这个接口非常必要；

15. 特定机型硬解码：特定机型硬解码，也主要是用于多路播放场景下，通过硬解码，实现更低的CPU占用目的；

16. 只播放关键帧：特别是大屏多实例场景播放的时候，尽管我们已经CPU占用非常低了，如果只是查看大概的监控情景，实现更多路的播放，只播放关键帧是个非常好的功能点，如果需要原始帧播放，可以实时调节即可；

17. TCP-UDP设置：考虑到部分服务器或硬件设备或网络环境对TCP、UDP某一个支持的比较好，我们加了设置接口；

18. TCP-UDP自动切换：这个是更细力度的接口，比如默认设置了TCP模式，TCP模式下收不到数据，超时后，自动切换到UDP模式尝试，一般开源播放器不具备此功能；

19. RTSP超时时间设定：比如10-12秒收不到数据，自动重连，一般开源播放器支持不好。

技术实现

本文以大牛直播SDK的Windows平台C++的demo为例，探讨下RTMP、RTSP播放、录像、实时音量调节、快照等接口设计和处理：

模块初始化：

	GetSmartPlayerSDKAPI(&player_api_);

	if ( NT_ERC_OK != player_api_.Init(0, NULL) )
	{
		return FALSE;
	}

	is_support_h264_hardware_decoder_ = NT_ERC_OK == player_api_.IsSupportH264HardwareDecoder();
	is_support_h265_hardware_decoder_ = NT_ERC_OK == player_api_.IsSupportH265HardwareDecoder();

	if ( NT_ERC_OK != player_api_.Open(&player_handle_, NULL, 0, NULL) )
	{
		return FALSE;
	}

	player_api_.SetEventCallBack(player_handle_, GetSafeHwnd(), &NT_SP_SDKEventHandle);

其他参数初始化：

bool CSmartPlayerDlg::InitCommonSDKParam()
{
	ASSERT(!is_playing_);
	ASSERT(!is_recording_);

	if ( NULL == player_handle_ )
		return false;

	CString wbuffer_str;
	edit_buffer.GetWindowTextW(wbuffer_str);

	std::wstring_convert<std::codecvt_utf8<wchar_t> > conv;

	auto buffer_str = conv.to_bytes(wbuffer_str);

	player_api_.SetBuffer(player_handle_, atoi(buffer_str.c_str()));

	// 设置rtsp 超时时间
	player_api_.SetRtspTimeout(player_handle_, rtsp_conf_info_.timeout_);

	// 设置rtsp tcp模式，rtmp不使用, 可以不设置
	player_api_.SetRTSPTcpMode(player_handle_, rtsp_conf_info_.is_tcp_ ? 1 : 0);

	player_api_.SetRtspAutoSwitchTcpUdp(player_handle_, rtsp_conf_info_.is_tcp_udp_auto_switch_ ? 1 : 0);

	if ( btn_check_fast_startup_.GetCheck() == BST_CHECKED )
	{
		player_api_.SetFastStartup(player_handle_, 1);
	}
	else
	{
		player_api_.SetFastStartup(player_handle_, 0);
	}

	player_api_.SetReportDownloadSpeed(player_handle_, 1, 1);

	if (NT_ERC_OK != player_api_.SetURL(player_handle_, GetURL().c_str()))
	{
		return false;
	}

	connection_status_	= 0;
	buffer_status_		= 0;
	buffer_percent_		= 0;
	download_speed_		= -1;

	return true;
}

播放控制：

void CSmartPlayerDlg::OnBnClickedButtonPlay()
{
	if ( player_handle_ == NULL )
		return;

	CString btn_play_str;

	btn_play_.GetWindowTextW(btn_play_str);

	if ( btn_play_str == _T("播放") )
	{
		if ( !is_recording_ )
		{
			if ( !InitCommonSDKParam() )
			{
				AfxMessageBox(_T("设置参数错误!"));
				return;
			}
		}
	
		player_api_.SetVideoSizeCallBack(player_handle_, GetSafeHwnd(), SP_SDKVideoSizeHandle);

		bool is_support_d3d_render = false;
		NT_INT32 in_support_d3d_render = 0;

		if ( NT_ERC_OK == player_api_.IsSupportD3DRender(player_handle_,
			wrapper_render_wnd_.RenderWnd(), &in_support_d3d_render))
		{
			if ( 1 == in_support_d3d_render )
			{
				is_support_d3d_render = true;
			}
		}

		if ( is_support_d3d_render )
		{
			is_gdi_render_ = false;

			// 支持d3d绘制的话，就用D3D绘制
			player_api_.SetRenderWindow(player_handle_, wrapper_render_wnd_.RenderWnd());

			player_api_.SetRenderScaleMode(player_handle_, btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);
		}
		else
		{
			is_gdi_render_ = true;

			// 不支持D3D就让播放器吐出数据来，用GDI绘制

			wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);

			player_api_.SetVideoFrameCallBack(player_handle_, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32,
				GetSafeHwnd(), SM_SDKVideoFrameHandle);
		}

		if ( BST_CHECKED == btn_check_hardware_decoder_.GetCheck() )
		{
			player_api_.SetH264HardwareDecoder(player_handle_, is_support_h264_hardware_decoder_?1:0, 0);
			player_api_.SetH265HardwareDecoder(player_handle_, is_support_h265_hardware_decoder_?1:0, 0);
		}
		else
		{
			player_api_.SetH264HardwareDecoder(player_handle_, 0, 0);
			player_api_.SetH265HardwareDecoder(player_handle_, 0, 0);
		}

		player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);

		player_api_.SetLowLatencyMode(player_handle_, BST_CHECKED == btn_check_low_latency_.GetCheck() ? 1 : 0);

		player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 :0 );

		player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);

		player_api_.SetRotation(player_handle_, rotate_degrees_);

		player_api_.SetAudioVolume(player_handle_, slider_audio_volume_.GetPos());

		player_api_.SetUserDataCallBack(player_handle_, GetSafeHwnd(), NT_SP_SDKUserDataHandle);

		if (NT_ERC_OK != player_api_.StartPlay(player_handle_))
		{
			AfxMessageBox(_T("播放器失败!"));
			return;
		}

		btn_play_.SetWindowTextW(_T("停止"));
		is_playing_ = true;
	}
	else
	{
		StopPlayback();
	}
}

实时快照：

void CSmartPlayerDlg::OnBnClickedButtonCaptureImage()
{
	if ( capture_image_path_.empty() )
	{
		AfxMessageBox(_T("请先设置保存截图文件的目录! 点击截图左边的按钮设置!"));
		return;
	}

	if ( player_handle_ == NULL )
	{
		return;
	}

	if ( !is_playing_ )
	{
		return;
	}

	std::wostringstream ss;
	ss << capture_image_path_;

	if ( capture_image_path_.back() != L'\\' )
	{
		ss << L"\\";
	}

	SYSTEMTIME sysTime;
	::GetLocalTime(&sysTime);

	ss << L"SmartPlayer-"
		<< std::setfill(L'0') << std::setw(4) << sysTime.wYear
		<< std::setfill(L'0') << std::setw(2) << sysTime.wMonth
		<< std::setfill(L'0') << std::setw(2) << sysTime.wDay
		<< L"-"
		<< std::setfill(L'0') << std::setw(2) << sysTime.wHour
		<< std::setfill(L'0') << std::setw(2) << sysTime.wMinute
		<< std::setfill(L'0') << std::setw(2) << sysTime.wSecond;

	ss << L"-" << std::setfill(L'0') << std::setw(3) << sysTime.wMilliseconds
		<< L".png";

	std::wstring_convert<std::codecvt_utf8<wchar_t> > conv;

	auto val_str = conv.to_bytes(ss.str());

	auto ret = player_api_.CaptureImage(player_handle_, val_str.c_str(), NULL, &SM_SDKCaptureImageHandle);
	if (NT_ERC_OK == ret)
	{
		// 发送截图请求成功
	}
	else if (NT_ERC_SP_TOO_MANY_CAPTURE_IMAGE_REQUESTS == ret)
	{
		// 通知用户延时
		OutputDebugStringA("Too many capture image requests!!!\r\n");
	}
	else
	{
		// 其他失败
	}
}

只解关键帧：

void CSmartPlayerDlg::OnBnClickedCheckOnlyDecodeVideoKeyFrame()
{
	if (player_handle_ != NULL)
	{
		player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);
	}
}

设置等比例显示或全铺满模式：

void CSmartPlayerDlg::OnBnClickedCheckRenderScaleMode()
{
	if (player_handle_ != NULL)
	{
		if (!is_gdi_render_)
		{
			player_api_.SetRenderScaleMode(player_handle_, BST_CHECKED == btn_check_render_scale_mode_.GetCheck() ? 1 : 0);
		}
		else
		{
			wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);
		}
	}
}

视频view水平反转、垂直反转、旋转：

void CSmartPlayerDlg::OnBnClickedCheckFlipHorizontal()
{
	if (player_handle_ != NULL)
	{
		player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);
	}
}

void CSmartPlayerDlg::OnBnClickedCheckFlipVertical()
{
	if (player_handle_ != NULL)
	{
		player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 : 0);
	}
}

void CSmartPlayerDlg::OnBnClickedButtonRotation()
{
	rotate_degrees_ += 90;
	rotate_degrees_ = rotate_degrees_ % 360;

	if (0 == rotate_degrees_)
	{
		btn_rotation_.SetWindowText(_T("旋转90度"));
	}
	else if (90 == rotate_degrees_)
	{
		btn_rotation_.SetWindowText(_T("旋转180度"));
	}
	else if (180 == rotate_degrees_)
	{
		btn_rotation_.SetWindowText(_T("旋转270度"));
	}
	else if (270 == rotate_degrees_)
	{
		btn_rotation_.SetWindowText(_T("不旋转"));
	}

	if ( player_handle_ != NULL )
	{
		player_api_.SetRotation(player_handle_, rotate_degrees_);
	}
}

实时录像：

void CSmartPlayerDlg::OnBnClickedButtonRecord()
{
	if ( player_handle_ == NULL )
		return;

	CString btn_record_str;
	btn_record_.GetWindowTextW(btn_record_str);

	if ( btn_record_str == _T("录像") )
	{
		if ( !rec_conf_info_.is_record_video_ && !rec_conf_info_.is_record_audio_ )
		{
			AfxMessageBox(_T("音频录制选项和视频录制选项至少需要选择一个!"));
			return;
		}

		if ( !is_playing_ )
		{
			if ( !InitCommonSDKParam() )
			{
				AfxMessageBox(_T("设置参数错误!"));
				return;
			}
		}

		player_api_.SetRecorderVideo(player_handle_, rec_conf_info_.is_record_video_ ? 1 : 0);
		player_api_.SetRecorderAudio(player_handle_, rec_conf_info_.is_record_audio_ ? 1 : 0);

		auto ret = player_api_.SetRecorderDirectory(player_handle_, rec_conf_info_.dir_.c_str());
		if ( NT_ERC_OK != ret )
		{
			AfxMessageBox(_T("设置录像目录失败，请确保目录存在且是英文目录"));
			return;
		}

		player_api_.SetRecorderFileMaxSize(player_handle_, rec_conf_info_.file_max_size_);

		NT_SP_RecorderFileNameRuler rec_name_ruler = { 0 };

		rec_name_ruler.type_ = 0;
		rec_name_ruler.file_name_prefix_ = rec_conf_info_.file_name_prefix_.c_str();
		rec_name_ruler.append_date_		 = rec_conf_info_.is_append_date_ ? 1 : 0;
		rec_name_ruler.append_time_		 = rec_conf_info_.is_append_time_ ? 1 : 0;

		player_api_.SetRecorderFileNameRuler(player_handle_, &rec_name_ruler);

		player_api_.SetRecorderCallBack(player_handle_, GetSafeHwnd(), &SP_SDKRecorderHandle);

		player_api_.SetRecorderAudioTranscodeAAC(player_handle_, rec_conf_info_.is_audio_transcode_aac_ ? 1 : 0);

		if ( NT_ERC_OK != player_api_.StartRecorder(player_handle_) )
		{
			AfxMessageBox(_T("录像失败!"));
			return;
		}

		btn_record_.SetWindowTextW(_T("停止录像"));
		is_recording_ = true;
	}
	else
	{
		StopRecorder();
	}
}

Event回调：

LRESULT CSmartPlayerDlg::OnSDKEvent(WPARAM wParam, LPARAM lParam)
{
	if (!is_playing_ && !is_recording_)
	{
		return S_OK;
	}

	NT_UINT32 event_id = (NT_UINT32)(wParam);

	if ( NT_SP_E_EVENT_ID_PLAYBACK_REACH_EOS == event_id )
	{
		StopPlayback();
		return S_OK;
	}
	else if ( NT_SP_E_EVENT_ID_RECORDER_REACH_EOS == event_id )
	{
		StopRecorder();
		return S_OK;
	}
	else if ( NT_SP_E_EVENT_ID_RTSP_STATUS_CODE == event_id )
	{
		int status_code = (int)lParam;
		if ( 401 == status_code )
		{
			HandleVerification();
		}

		return S_OK;
	}
	else if (NT_SP_E_EVENT_ID_NEED_KEY == event_id)
	{
		HandleKeyEvent(false);

		return S_OK;
	}
	else if (NT_SP_E_EVENT_ID_KEY_ERROR == event_id)
	{
		HandleKeyEvent(true);

		return S_OK;
	}
	else if ( NT_SP_E_EVENT_ID_PULLSTREAM_REACH_EOS == event_id )
	{
		if (player_handle_ != NULL)
		{
			player_api_.StopPullStream(player_handle_);
		}

		return S_OK;
	}
	else if ( NT_SP_E_EVENT_ID_DURATION == event_id )
	{
		NT_INT64 duration = (NT_INT64)(lParam);

		edit_duration_.SetWindowTextW(GetHMSMsFormatStr(duration, false, false).c_str());

		return S_OK;
	}

	if ( NT_SP_E_EVENT_ID_CONNECTING == event_id
		|| NT_SP_E_EVENT_ID_CONNECTION_FAILED == event_id
		|| NT_SP_E_EVENT_ID_CONNECTED == event_id
		|| NT_SP_E_EVENT_ID_DISCONNECTED == event_id
		|| NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == event_id)
	{
		if ( NT_SP_E_EVENT_ID_CONNECTING == event_id )
		{
			OutputDebugStringA("connection status: connecting\r\n");
		}
		else if ( NT_SP_E_EVENT_ID_CONNECTION_FAILED == event_id )
		{
			OutputDebugStringA("connection status: connection failed\r\n");
		}
		else if ( NT_SP_E_EVENT_ID_CONNECTED == event_id )
		{
			OutputDebugStringA("connection status: connected\r\n");
		}
		else if (NT_SP_E_EVENT_ID_DISCONNECTED == event_id)
		{
			OutputDebugStringA("connection status: disconnected\r\n");
		}
		else if (NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == event_id)
		{
			OutputDebugStringA("connection status: no mediadata received\r\n");
		}

		connection_status_ = event_id;
	}

	if ( NT_SP_E_EVENT_ID_START_BUFFERING == event_id
		|| NT_SP_E_EVENT_ID_BUFFERING == event_id
		|| NT_SP_E_EVENT_ID_STOP_BUFFERING == event_id )
	{
		buffer_status_ = event_id;
		
		if ( NT_SP_E_EVENT_ID_BUFFERING == event_id )
		{
			buffer_percent_ = (NT_INT32)lParam;

			std::wostringstream ss;
			ss << L"buffering:" << buffer_percent_ << "%";
			OutputDebugStringW(ss.str().c_str());
			OutputDebugStringW(L"\r\n");
		}
	}

	if ( NT_SP_E_EVENT_ID_DOWNLOAD_SPEED == event_id )
	{
		download_speed_ = (NT_INT32)lParam;
	}

	CString show_str = base_title_;

	if ( connection_status_ != 0 )
	{
		show_str += _T("--链接状态: ");

		if ( NT_SP_E_EVENT_ID_CONNECTING == connection_status_ )
		{
			show_str += _T("链接中");
		}
		else if ( NT_SP_E_EVENT_ID_CONNECTION_FAILED == connection_status_ )
		{
			show_str += _T("链接失败");
		}
		else if ( NT_SP_E_EVENT_ID_CONNECTED == connection_status_ )
		{
			show_str += _T("链接成功");
		}
		else if ( NT_SP_E_EVENT_ID_DISCONNECTED == connection_status_ )
		{
			show_str += _T("链接断开");
		}
		else if (NT_SP_E_EVENT_ID_NO_MEDIADATA_RECEIVED == connection_status_)
		{
			show_str += _T("收不到数据");
		}
	}

	if (download_speed_ != -1)
	{
		std::wostringstream ss;
		ss << L"--下载速度:" << (download_speed_ * 8 / 1000) << "kbps"
		  << L"(" << (download_speed_ / 1024) << "KB/s)";

		show_str += ss.str().c_str();
	}

	if ( buffer_status_ != 0 )
	{
		show_str += _T("--缓冲状态: ");

		if ( NT_SP_E_EVENT_ID_START_BUFFERING == buffer_status_ )
		{
			show_str += _T("开始缓冲");
		}
		else if (NT_SP_E_EVENT_ID_BUFFERING == buffer_status_)
		{
			std::wostringstream ss;
			ss << L"缓冲中" << buffer_percent_ << "%";
			show_str += ss.str().c_str();
		}
		else if (NT_SP_E_EVENT_ID_STOP_BUFFERING == buffer_status_)
		{
			show_str += _T("结束缓冲");
		}
	}


	SetWindowText(show_str);

	return S_OK;
}