目录
- 1.视频流编码器（VideoStreamEncoder）
- 1.1 视频流编码器接口（VideoStreamEncoderInterface）
- 1.2 资源类（Resource）
- 1.3 已编码图像的回调（EncodedImageCallback）
- 1.4 视频资源监听器（VideoSourceRestrictionsListener）
1.视频流编码器（VideoStreamEncoder）
VideoStreamEncoder是编码流程当中至关重要的一个类，它实现的功能包括：
（1）开始编码器，暂停编码器，停止编码器
（2）设置编码器（根据config）
（3）调控编码参数，如码控参数，帧率等等
（4）调控前向纠错器
（5）视频帧的传输和接收
（6）根据网络环境，调控丢帧
（7）根据编码情况，调控屯帧
（8）根据情况，调控编码质量（因为网络环境，编码器情况，可能会调整编码策略，影响编码质量）
（9）对已编码图像进行后处理
（10）统计编码信息
可以说，在视频编码流程中，VideoStreamEncoder是获取了视频帧之后，第一个会使用到的类。这个类十分综合，大多数的编码过程都与这个类相关，所以其声明比较长，这里做简单记录，深入到每个模块之后再分析。
// VideoStreamEncoder represent a video encoder that accepts raw video frames as
// input and produces an encoded bit stream.
// Usage:
// Instantiate.
// Call SetSink.
// Call SetSource.
// Call ConfigureEncoder with the codec settings.
// Call Stop() when done.
/*
VideoStreamEncoder表示了一个视频编码器，它接收原始视频帧作为输入并且产生编码后的比特流
用法:
(1) 创建实例
(2) SetSink，设置接收器
(3) SetSource，设置资源
(4) ConfigureEncoder，配置编码器参数
(5) Stop()，结束流程
*/
class VideoStreamEncoder : public VideoStreamEncoderInterface,
private EncodedImageCallback,
public VideoSourceRestrictionsListener {
public:
// TODO(bugs.webrtc.org/12000): Reporting of VideoBitrateAllocation is being
// deprecated. Instead VideoLayersAllocation should be reported.
enum class BitrateAllocationCallbackType {
kVideoBitrateAllocation,
kVideoBitrateAllocationWhenScreenSharing,
kVideoLayersAllocation
};
VideoStreamEncoder(
const Environment& env,
uint32_t number_of_cores,
VideoStreamEncoderObserver* encoder_stats_observer,
const VideoStreamEncoderSettings& settings,
std::unique_ptr<OveruseFrameDetector> overuse_detector,
std::unique_ptr<FrameCadenceAdapterInterface> frame_cadence_adapter,
std::unique_ptr<webrtc::TaskQueueBase, webrtc::TaskQueueDeleter>
encoder_queue,
BitrateAllocationCallbackType allocation_cb_type,
webrtc::VideoEncoderFactory::EncoderSelectorInterface* encoder_selector =
nullptr);
~VideoStreamEncoder() override;
VideoStreamEncoder(const VideoStreamEncoder&) = delete;
VideoStreamEncoder& operator=(const VideoStreamEncoder&) = delete;
// 添加自适应资源
void AddAdaptationResource(rtc::scoped_refptr<Resource> resource) override;
// 获取自适应资源
std::vector<rtc::scoped_refptr<Resource>> GetAdaptationResources() override;
// 设置资源
void SetSource(rtc::VideoSourceInterface<VideoFrame>* source,
const DegradationPreference& degradation_preference) override;
// 设置接收器
void SetSink(EncoderSink* sink, bool rotation_applied) override;
// TODO(perkj): Can we remove VideoCodec.startBitrate ?
// 设置起始码率
void SetStartBitrate(int start_bitrate_bps) override;
// 设置前向纠错控制器
void SetFecControllerOverride(
FecControllerOverride* fec_controller_override) override;
// 配置编码器
void ConfigureEncoder(VideoEncoderConfig config,
size_t max_data_payload_length) override;
void ConfigureEncoder(VideoEncoderConfig config,
size_t max_data_payload_length,
SetParametersCallback callback) override;
// Permanently stop encoding. After this method has returned, it is
// guaranteed that no encoded frames will be delivered to the sink.
// 永久性停止编码，此方法返回后，保证不会有编码后的帧传递给接收器
void Stop() override;
// 传递关键帧
void SendKeyFrame(const std::vector<VideoFrameType>& layers = {}) override;
// 检测到丢包，需要调整编码策略，例如降低分辨率或帧率
void OnLossNotification(
const VideoEncoder::LossNotification& loss_notification) override;
// 检测到码率变化，需要调整编码参数
void OnBitrateUpdated(DataRate target_bitrate,
DataRate stable_target_bitrate,
DataRate target_headroom,
uint8_t fraction_lost,
int64_t round_trip_time_ms,
double cwnd_reduce_ratio) override;
// 根据拥塞窗口减少比例，来调整目标比特率，避免丢包和网络拥塞
DataRate UpdateTargetBitrate(DataRate target_bitrate,
double cwnd_reduce_ratio);
protected:
friend class VideoStreamEncoderFrameCadenceRestrictionTest;
// Used for testing. For example the `ScalingObserverInterface` methods must
// be called on `encoder_queue_`.
TaskQueueBase* encoder_queue() { return encoder_queue_.get(); }
// 检测到视频限制性资源发生变化
void OnVideoSourceRestrictionsUpdated(
VideoSourceRestrictions restrictions,
const VideoAdaptationCounters& adaptation_counters,
rtc::scoped_refptr<Resource> reason,
const VideoSourceRestrictions& unfiltered_restrictions) override;
// Used for injected test resources.
// TODO(eshr): Move all adaptation tests out of VideoStreamEncoder tests.
// 注入自适应资源，这里与AddAdaptationResource不同，还需要入参reason，即更加
// 精准的调控
void InjectAdaptationResource(rtc::scoped_refptr<Resource> resource,
VideoAdaptationReason reason);
// 注入自适应限制
void InjectAdaptationConstraint(AdaptationConstraint* adaptation_constraint);
void AddRestrictionsListenerForTesting(
VideoSourceRestrictionsListener* restrictions_listener);
void RemoveRestrictionsListenerForTesting(
VideoSourceRestrictionsListener* restrictions_listener);
private:
class CadenceCallback : public FrameCadenceAdapterInterface::Callback {
public:
explicit CadenceCallback(VideoStreamEncoder& video_stream_encoder)
: video_stream_encoder_(video_stream_encoder) {}
// FrameCadenceAdapterInterface::Callback overrides.
// 下面是对FrameCadenceAdapterInterface::Callback当中函数的重载
void OnFrame(Timestamp post_time,
bool queue_overload,
const VideoFrame& frame) override {
// 使用VideoStreamEncoder当中的OnFrame()函数重载
video_stream_encoder_.OnFrame(post_time, queue_overload, frame);
}
void OnDiscardedFrame() override {
video_stream_encoder_.OnDiscardedFrame();
}
void RequestRefreshFrame() override {
video_stream_encoder_.RequestRefreshFrame();
}
private:
VideoStreamEncoder& video_stream_encoder_;
};
// VideoFrameInfo记录长宽，是否是纹理等信息
class VideoFrameInfo {
public:
VideoFrameInfo(int width, int height, bool is_texture)
: width(width), height(height), is_texture(is_texture) {}
int width;
int height;
bool is_texture;
int pixel_count() const { return width * height; }
};
// 编码速率设定
struct EncoderRateSettings {
EncoderRateSettings();
EncoderRateSettings(const VideoBitrateAllocation& bitrate,
double framerate_fps,
DataRate bandwidth_allocation,
DataRate encoder_target,
DataRate stable_encoder_target);
bool operator==(const EncoderRateSettings& rhs) const;
bool operator!=(const EncoderRateSettings& rhs) const;
// 码率控制参数
VideoEncoder::RateControlParameters rate_control;
// This is the scalar target bitrate before the VideoBitrateAllocator, i.e.
// the `target_bitrate` argument of the OnBitrateUpdated() method. This is
// needed because the bitrate allocator may truncate the total bitrate and a
// later call to the same allocator instance, e.g.
// |using last_encoder_rate_setings_->bitrate.get_sum_bps()|, may trick it
// into thinking the available bitrate has decreased since the last call.
/*
这是`VideoBitrateAllocator`之前的标量目标比特率，即`OnBitrateUpdated()`方法
的`target_bitrate`参数。这是因为比特率分配器可能会截断总比特率，而在稍后对同一
分配器实例的调用中，例如`|using last_encoder_rate_setings_->bitrate.get_sum_bps()|`，
可能会误导它认为自上次调用以来可用的比特率已经减少。
*/
DataRate encoder_target;
DataRate stable_encoder_target;
};
class DegradationPreferenceManager;
// 重新配置编码器
void ReconfigureEncoder() RTC_RUN_ON(encoder_queue_);
// 检测到编码器配置发生变化
void OnEncoderSettingsChanged() RTC_RUN_ON(encoder_queue_);
// 检测到帧
void OnFrame(Timestamp post_time,
bool queue_overload,
const VideoFrame& video_frame);
// 检测到丢弃帧
void OnDiscardedFrame();
// 需要刷新帧
void RequestRefreshFrame();
// 可能会编码视频帧
void MaybeEncodeVideoFrame(const VideoFrame& frame,
int64_t time_when_posted_in_ms);
// 编码视频帧
void EncodeVideoFrame(const VideoFrame& frame,
int64_t time_when_posted_in_ms);
// Indicates whether frame should be dropped because the pixel count is too
// large for the current bitrate configuration.
// 在当前码率配置下，根据pixle count来检查当前帧是否需要被丢弃
bool DropDueToSize(uint32_t pixel_count) const RTC_RUN_ON(encoder_queue_);
// Implements EncodedImageCallback.
// 接收编码器编码之后的图像
EncodedImageCallback::Result OnEncodedImage(
const EncodedImage& encoded_image,
const CodecSpecificInfo* codec_specific_info) override;
// 丢帧的回调函数，需要给出reason
void OnDroppedFrame(EncodedImageCallback::DropReason reason) override;
// 暂停编码器
bool EncoderPaused() const;
// 标记丢帧开始
void TraceFrameDropStart();
// 标记丢帧结束
void TraceFrameDropEnd();
// Returns a copy of `rate_settings` with the `bitrate` field updated using
// the current VideoBitrateAllocator.
// 返回一个`rate_settings`的副本，其中`bitrate`字段已使用当前的`VideoBitrateAllocator`更新。
EncoderRateSettings UpdateBitrateAllocation(
const EncoderRateSettings& rate_settings) RTC_RUN_ON(encoder_queue_);
// 获取输入帧率Fps
uint32_t GetInputFramerateFps() RTC_RUN_ON(encoder_queue_);
// 设置编码器速率
void SetEncoderRates(const EncoderRateSettings& rate_settings)
RTC_RUN_ON(encoder_queue_);
// 进行编码后处理
void RunPostEncode(const EncodedImage& encoded_image,
int64_t time_sent_us,
int temporal_index,
DataSize frame_size);
// 释放编码器
void ReleaseEncoder() RTC_RUN_ON(encoder_queue_);
// After calling this function `resource_adaptation_processor_` will be null.
// 关闭资源自适应队列，resource_adaptation_processor会设置为null
void ShutdownResourceAdaptationQueue();
// 申请更换编码器
void RequestEncoderSwitch() RTC_RUN_ON(encoder_queue_);
// Augments an EncodedImage received from an encoder with parsable
// information.
// 修改或增强已编码图像
EncodedImage AugmentEncodedImage(
const EncodedImage& encoded_image,
const CodecSpecificInfo* codec_specific_info);
// 处理已丢弃的帧
void ProcessDroppedFrame(const VideoFrame& frame,
VideoStreamEncoderObserver::DropReason reason)
RTC_RUN_ON(encoder_queue_);
const Environment env_;
TaskQueueBase* const worker_queue_;
const int number_of_cores_;
// sink是接收器
EncoderSink* sink_ = nullptr;
// VideoStreamEncoder配置
const VideoStreamEncoderSettings settings_;
// 比特分配回调的类型
const BitrateAllocationCallbackType allocation_cb_type_;
// 码率控制配置
const RateControlSettings rate_control_settings_;
webrtc::VideoEncoderFactory::EncoderSelectorInterface* const
encoder_selector_from_constructor_;
std::unique_ptr<VideoEncoderFactory::EncoderSelectorInterface> const
encoder_selector_from_factory_;
// Pointing to either encoder_selector_from_constructor_ or
// encoder_selector_from_factory_ but can be nullptr.
// 编码器选择器
VideoEncoderFactory::EncoderSelectorInterface* const encoder_selector_;
// 视频流编码器观察者的行为
VideoStreamEncoderObserver* const encoder_stats_observer_;
// Adapter that avoids public inheritance of the cadence adapter's callback
// interface.
// 适配器，避免了公开继承节拍适配器的回调接口。
CadenceCallback cadence_callback_{*this};
// Frame cadence encoder adapter. Frames enter this adapter first, and it then
// forwards them to our OnFrame method.
// 帧节拍编码器适配器。帧首先进入这个适配器，然后它将帧转发到我们的OnFrame方法。
std::unique_ptr<FrameCadenceAdapterInterface> frame_cadence_adapter_
RTC_GUARDED_BY(encoder_queue_) RTC_PT_GUARDED_BY(encoder_queue_);
// 编码器配置项
VideoEncoderConfig encoder_config_ RTC_GUARDED_BY(encoder_queue_);
// 编码器
std::unique_ptr<VideoEncoder> encoder_ RTC_GUARDED_BY(encoder_queue_)
RTC_PT_GUARDED_BY(encoder_queue_);
bool encoder_initialized_ = false;
// 码率分配器
std::unique_ptr<VideoBitrateAllocator> rate_allocator_
RTC_GUARDED_BY(encoder_queue_) RTC_PT_GUARDED_BY(encoder_queue_);
int max_framerate_ RTC_GUARDED_BY(encoder_queue_) = -1;
// Set when ConfigureEncoder has been called in order to lazy reconfigure the
// encoder on the next frame.
// 当调用了ConfigureEncoder时设置，以便在下一帧上延迟重新配置编码器。
bool pending_encoder_reconfiguration_ RTC_GUARDED_BY(encoder_queue_) = false;
// Set when configuration must create a new encoder object, e.g.,
// because of a codec change.
// 当配置必须创建一个新的编码器对象时设置，例如，由于编解码器变更。
bool pending_encoder_creation_ RTC_GUARDED_BY(encoder_queue_) = false;
// 编码器配置的回调函数
absl::InlinedVector<SetParametersCallback, 2> encoder_configuration_callbacks_
RTC_GUARDED_BY(encoder_queue_);
// 上一帧的信息
std::optional<VideoFrameInfo> last_frame_info_ RTC_GUARDED_BY(encoder_queue_);
// 边界裁剪的width和height
int crop_width_ RTC_GUARDED_BY(encoder_queue_) = 0;
int crop_height_ RTC_GUARDED_BY(encoder_queue_) = 0;
// 编码器目标比特率
std::optional<uint32_t> encoder_target_bitrate_bps_
RTC_GUARDED_BY(encoder_queue_);
// 最大数据负载长度
size_t max_data_payload_length_ RTC_GUARDED_BY(encoder_queue_) = 0;
// 上一个编码器的码率设定
std::optional<EncoderRateSettings> last_encoder_rate_settings_
RTC_GUARDED_BY(encoder_queue_);
// 编码器是否停止并丢帧
bool encoder_paused_and_dropped_frame_ RTC_GUARDED_BY(encoder_queue_) = false;
// Set to true if at least one frame was sent to encoder since last encoder
// initialization.
// 如果自上次编码器初始化以来至少有一帧已发送到编码器，则设置为true
bool was_encode_called_since_last_initialization_
RTC_GUARDED_BY(encoder_queue_) = false;
// 编码器是否失败
bool encoder_failed_ RTC_GUARDED_BY(encoder_queue_) = false;
// Used to make sure incoming time stamp is increasing for every frame.
// 用于确保每个帧的到达时间戳都在增加。
int64_t last_captured_timestamp_ RTC_GUARDED_BY(encoder_queue_) = 0;
// Delta used for translating between NTP and internal timestamps.
// 用于在NTP时间戳和内部时间戳之间转换的差值。
const int64_t delta_ntp_internal_ms_ RTC_GUARDED_BY(encoder_queue_);
// 最后一次记录帧日志的时间戳
int64_t last_frame_log_ms_ RTC_GUARDED_BY(encoder_queue_);
// 已捕捉帧的数量
int captured_frame_count_ RTC_GUARDED_BY(encoder_queue_) = 0;
// 由于网络阻塞丢弃的帧数量
int dropped_frame_cwnd_pushback_count_ RTC_GUARDED_BY(encoder_queue_) = 0;
// 由于编码器阻塞丢弃的帧数量
int dropped_frame_encoder_block_count_ RTC_GUARDED_BY(encoder_queue_) = 0;
// 屯帧的数量（待编码帧数量）
std::optional<VideoFrame> pending_frame_ RTC_GUARDED_BY(encoder_queue_);
// 屯帧的时间
int64_t pending_frame_post_time_us_ RTC_GUARDED_BY(encoder_queue_) = 0;
// 存储了视频帧中自上次更新以来发生变化的区域
VideoFrame::UpdateRect accumulated_update_rect_
RTC_GUARDED_BY(encoder_queue_);
// 指示accumulated_update_rect_是否包含有效的更新区域信息
bool accumulated_update_rect_is_valid_ RTC_GUARDED_BY(encoder_queue_) = true;
// 前向纠错控制
FecControllerOverride* fec_controller_override_
RTC_GUARDED_BY(encoder_queue_) = nullptr;
// 记录最近一次参数更新的时间
std::optional<int64_t> last_parameters_update_ms_
RTC_GUARDED_BY(encoder_queue_);
// 记录最近一次编码操作的时间戳
std::optional<int64_t> last_encode_info_ms_ RTC_GUARDED_BY(encoder_queue_);
// 编码器信息
VideoEncoder::EncoderInfo encoder_info_ RTC_GUARDED_BY(encoder_queue_);
// 编码器，存储视频编解码器的相关配置信息，包括编解码器类型、编码设置、分辨率、帧率、比特率等参数
VideoCodec send_codec_ RTC_GUARDED_BY(encoder_queue_);
// 帧丢弃器
FrameDropper frame_dropper_ RTC_GUARDED_BY(encoder_queue_);
// If frame dropper is not force disabled, frame dropping might still be
// disabled if VideoEncoder::GetEncoderInfo() indicates that the encoder has a
// trusted rate controller. This is determined on a per-frame basis, as the
// encoder behavior might dynamically change.
/*
如果帧丢弃器没有被强制禁用，那么即使 VideoEncoder::GetEncoderInfo() 表示编码器
拥有一个可信的速率控制器，帧丢弃可能仍然被禁用。这是逐帧决定的，因为编码器的行为可能会动态变化。
*/
// 强制禁用frameDropper
bool force_disable_frame_dropper_ RTC_GUARDED_BY(encoder_queue_) = false;
// Incremented on worker thread whenever `frame_dropper_` determines that a
// frame should be dropped. Decremented on whichever thread runs
// OnEncodedImage(), which is only called by one thread but not necessarily
// the worker thread.
/*
每当 `frame_dropper_` 确定应该丢弃一帧时，在工作线程上增加。在运行 `OnEncodedImage()`
的线程上减少，这个函数只由一个线程调用，但不一定是工作线程。
*/
std::atomic<int> pending_frame_drops_{0};
// Congestion window frame drop ratio (drop 1 in every
// cwnd_frame_drop_interval_ frames).
// 拥塞窗口帧丢弃比例（每 cwnd_frame_drop_interval_ 帧丢弃 1 帧）。
std::optional<int> cwnd_frame_drop_interval_ RTC_GUARDED_BY(encoder_queue_);
// Frame counter for congestion window frame drop.
// 用于拥塞窗口帧丢弃的帧计数器
int cwnd_frame_counter_ RTC_GUARDED_BY(encoder_queue_) = 0;
// 码率调节器
std::unique_ptr<EncoderBitrateAdjuster> bitrate_adjuster_
RTC_GUARDED_BY(encoder_queue_);
// TODO(sprang): Change actually support keyframe per simulcast stream, or
// turn this into a simple bool `pending_keyframe_request_`.
// 下一帧的类型
std::vector<VideoFrameType> next_frame_types_ RTC_GUARDED_BY(encoder_queue_);
// 用于处理和写入与视频帧编码相关的元数据
FrameEncodeMetadataWriter frame_encode_metadata_writer_{this};
// Provides video stream input states: current resolution and frame rate.
// 处理视频流输入状态
VideoStreamInputStateProvider input_state_provider_;
// 视频流适应器，用于适配不同的视频流需求，包括分辨率、帧率和比特率
const std::unique_ptr<VideoStreamAdapter> video_stream_adapter_
RTC_GUARDED_BY(encoder_queue_);
// Responsible for adapting input resolution or frame rate to ensure resources
// (e.g. CPU or bandwidth) are not overused. Adding resources can occur on any
// thread.
// 负责调整输入分辨率或帧率，以确保资源（如CPU或带宽）不被过度使用。增加资源可以在任何线程上进行。
// 资源自适应处理器
std::unique_ptr<ResourceAdaptationProcessorInterface>
resource_adaptation_processor_ RTC_GUARDED_BY(encoder_queue_);
// 管理质量下降的偏好设置，因为在调控编码策略时，很有可能会带来视频质量的下降，例如快速算法等
std::unique_ptr<DegradationPreferenceManager> degradation_preference_manager_
RTC_GUARDED_BY(encoder_queue_);
// 自适应限制
std::vector<AdaptationConstraint*> adaptation_constraints_
RTC_GUARDED_BY(encoder_queue_);
// Handles input, output and stats reporting related to VideoStreamEncoder
// specific resources, such as "encode usage percent" measurements and "QP
// scaling". Also involved with various mitigations such as initial frame
// dropping.
// The manager primarily operates on the `encoder_queue_` but its lifetime is
// tied to the VideoStreamEncoder (which is destroyed off the encoder queue)
// and its resource list is accessible from any thread.
/*
处理与VideoStreamEncoder相关的输入、输出和统计报告，特别是“编码使用百分比”测量和
“QP缩放”等资源。还涉及各种缓解措施，如初始帧丢弃。该管理器主要在`encoder_queue_`
上操作，但其生命周期与VideoStreamEncoder（在编码器队列外销毁）相关联，其资源列表
可以从任何线程访问。
*/
VideoStreamEncoderResourceManager stream_resource_manager_
RTC_GUARDED_BY(encoder_queue_);
// 额外的资源
std::vector<rtc::scoped_refptr<Resource>> additional_resources_
RTC_GUARDED_BY(encoder_queue_);
// Carries out the VideoSourceRestrictions provided by the
// ResourceAdaptationProcessor, i.e. reconfigures the source of video frames
// to provide us with different resolution or frame rate.
// This class is thread-safe.
/*
执行由ResourceAdaptationProcessor提供的VideoSourceRestrictions，
即重新配置视频帧的源，为我们提供不同的分辨率或帧率。这个类是线程安全的。
*/
VideoSourceSinkController video_source_sink_controller_
RTC_GUARDED_BY(worker_queue_);
// Default bitrate limits in EncoderInfoSettings allowed.
// 默认的比特率限制
const bool default_limits_allowed_;
// QP parser is used to extract QP value from encoded frame when that is not
// provided by encoder.
// QP解析器用于在编码器未提供QP值时，从编码帧中提取QP值。
QpParser qp_parser_;
const bool qp_parsing_allowed_;
// The quality convergence controller is used to determine if a codec has
// reached its target quality. This is used for screenshare to determine when
// there's no need to continue encoding the same repeated frame.
/*
质量收敛控制器用于确定一个编解码器是否已经达到了其目标质量。
这在屏幕共享中用于判断何时不需要继续编码相同的重复帧。
*/
QualityConvergenceController quality_convergence_controller_
RTC_GUARDED_BY(encoder_queue_);
// Enables encoder switching on initialization failures.
// 在初始化失败时启用编码器切换。
bool switch_encoder_on_init_failures_;
const std::optional<int> vp9_low_tier_core_threshold_;
const std::optional<int> experimental_encoder_thread_limit_;
// This is a copy of restrictions (glorified max_pixel_count) set by
// OnVideoSourceRestrictionsUpdated. It is used to scale down encoding
// resolution if needed when using requested_resolution.
//
// TODO(webrtc:14451) Split video_source_sink_controller_
// so that ownership on restrictions/wants is kept on &encoder_queue_, that
// these extra copies would not be needed.
/*
这是由OnVideoSourceRestrictionsUpdated设置的限制（实际上是最大像素计数）的副本。
当使用requested_resolution时，如果需要，它被用来降低编码分辨率。
TODO(webrtc:14451) 分离video_source_sink_controller_，以便将限制/需求的所有权
保留在&encoder_queue_上，这样就不需要这些额外的副本了。
*/
std::optional<VideoSourceRestrictions> latest_restrictions_
RTC_GUARDED_BY(encoder_queue_);
// Used to cancel any potentially pending tasks to the worker thread.
// Refrenced by tasks running on `encoder_queue_` so need to be destroyed
// after stopping that queue. Must be created and destroyed on
// `worker_queue_`.
/*
用于取消任何可能挂起的工作任务线程任务。被在`encoder_queue_`上运行的任务引用，
因此需要在停止该队列后销毁。必须在`worker_queue_`上创建和销毁。
*/
ScopedTaskSafety task_safety_;
// 编码队列
std::unique_ptr<TaskQueueBase, TaskQueueDeleter> encoder_queue_;
// Required for automatic corruption detection.
// 自动检测数据损坏
std::unique_ptr<FrameInstrumentationGenerator>
frame_instrumentation_generator_;
};
1.1 视频流编码器接口（VideoStreamEncoderInterface）
VideoStreamEncoder的父类是VideoStreamEncoderInterface，这里定义了一些基础的函数。类的声明位于video/video_stream_encoder_interface.h中
// This interface represents a class responsible for creating and driving the
// encoder(s) for a single video stream. It is also responsible for adaptation
// decisions related to video quality, requesting reduced frame rate or
// resolution from the VideoSource when needed.
// TODO(bugs.webrtc.org/8830): This interface is under development. Changes
// under consideration include:
//
// 1. Taking out responsibility for adaptation decisions, instead only reporting
// per-frame measurements to the decision maker.
//
// 2. Moving responsibility for simulcast and for software fallback into this
// class.
/*
这个接口代表一个负责创建和驱动单个视频流的编码器（encoder）的类。
它还负责与视频质量相关的适应性决策，例如在需要时从 VideoSource 请求降低帧率或分辨率。
TODO(bugs.webrtc.org/8830): 此接口正在开发中。正在考虑的变更包括：
1. 移除适应性决策的责任，改为仅向决策者报告每帧的测量数据。
2. 将simulcast（同时向多个接收者发送不同质量的视频流）和软件回退的责任转移到这个类中。
*/
class VideoStreamEncoderInterface {
public:
// Interface for receiving encoded video frames and notifications about
// configuration changes.
// 接收编码后的视频帧以及关于配置变更通知的接口
class EncoderSink : public EncodedImageCallback {
public:
// 检测到编码器配置发生变化
virtual void OnEncoderConfigurationChanged(
std::vector<VideoStream> streams,
bool is_svc,
VideoEncoderConfig::ContentType content_type,
int min_transmit_bitrate_bps) = 0;
// 检测到码率分配发生变化
virtual void OnBitrateAllocationUpdated(
const VideoBitrateAllocation& allocation) = 0;
// 检测到视频层级分配发生变化
virtual void OnVideoLayersAllocationUpdated(
VideoLayersAllocation allocation) = 0;
};
virtual ~VideoStreamEncoderInterface() = default;
// If the resource is overusing, the VideoStreamEncoder will try to reduce
// resolution or frame rate until no resource is overusing.
// TODO(https://crbug.com/webrtc/11565): When the ResourceAdaptationProcessor
// is moved to Call this method could be deleted altogether in favor of
// Call-level APIs only
// 如果资源使用过量，VideoStreamEncoder 将尝试降低分辨率或帧率，直到没有资源使用过量为止。
// TODO(https://crbug.com/webrtc/11565): 当 ResourceAdaptationProcessor 被移动到
// Call 时，这个方法可能会被完全删除，转而只使用 Call 级别的 API。
virtual void AddAdaptationResource(rtc::scoped_refptr<Resource> resource) = 0;
virtual std::vector<rtc::scoped_refptr<Resource>>
GetAdaptationResources() = 0;
// Sets the source that will provide video frames to the VideoStreamEncoder's
// OnFrame method. `degradation_preference` control whether or not resolution
// or frame rate may be reduced. The VideoStreamEncoder registers itself with
// `source`, and signals adaptation decisions to the source in the form of
// VideoSinkWants.
// TODO(bugs.webrtc.org/14246): When adaptation logic is extracted from this
// class, it no longer needs to know the source.
/*
设置提供视频帧给 VideoStreamEncoder 的 OnFrame 方法的源。`degradation_preference`
控制是否允许降低分辨率或帧率。VideoStreamEncoder 在 `source` 中注册自己，并以
VideoSinkWants 的形式向源发出适应性决策信号。
TODO(bugs.webrtc.org/14246): 当适应性逻辑从这个类中提取出来时，它就不再需要知道源。
*/
virtual void SetSource(
rtc::VideoSourceInterface<VideoFrame>* source,
const DegradationPreference& degradation_preference) = 0;
// Sets the `sink` that gets the encoded frames. `rotation_applied` means
// that the source must support rotation. Only set `rotation_applied` if the
// remote side does not support the rotation extension.
/*
设置接收编码帧的 `sink`。`rotation_applied` 表示源必须支持旋转。
只有在远端不支持旋转扩展时才设置 `rotation_applied`。
*/
virtual void SetSink(EncoderSink* sink, bool rotation_applied) = 0;
// Sets an initial bitrate, later overriden by OnBitrateUpdated. Mainly
// affects the resolution of the initial key frame: If incoming frames are
// larger than reasonable for the start bitrate, and scaling is enabled,
// VideoStreamEncoder asks the source to scale down and drops a few initial
// frames.
// TODO(nisse): This is a poor interface, and mixes bandwidth estimation and
// codec configuration in an undesired way. For the actual send bandwidth, we
// should always be somewhat conservative, but we may nevertheless want to let
// the application configure a more optimistic quality for the initial
// resolution. Should be replaced by a construction time setting.
/*
设置初始比特率，后续会被 OnBitrateUpdated 方法覆盖。主要影响初始关键帧的分辨率：
如果传入的帧比初始比特率合理的大小要大，并且缩放被启用，VideoStreamEncoder
会请求源缩小尺寸并丢弃一些初始帧。
TODO(nisse): 这是一个设计不佳的接口，它以不期望的方式混合了带宽估计和编解码器配置。
对于实际的发送带宽，我们应该始终保持一定的保守性，但我们可能仍然希望允许应用程序为
初始分辨率配置一个更乐观的质量。应该被构造时的设置所取代。
*/
virtual void SetStartBitrate(int start_bitrate_bps) = 0;
// Request a key frame. Used for signalling from the remote receiver with
// no arguments and for RTCRtpSender.generateKeyFrame with a list of
// rids/layers.
/*
请求一个关键帧。用于从远程接收方发出信号，无需参数，
以及用于 RTCRtpSender.generateKeyFrame 方法，该方法带有一系列 rids/layers。
*/
virtual void SendKeyFrame(const std::vector<VideoFrameType>& layers = {}) = 0;
// Inform the encoder that a loss has occurred.
// 通知编码器发生了丢包
virtual void OnLossNotification(
const VideoEncoder::LossNotification& loss_notification) = 0;
// Set the currently estimated network properties. A `target_bitrate`
// of zero pauses the encoder.
// `stable_target_bitrate` is a filtered version of `target_bitrate`. It is
// always less or equal to it. It can be used to avoid rapid changes of
// expensive encoding settings, such as resolution.
// `link_allocation` is the bandwidth available for this video stream on the
// network link. It is always at least `target_bitrate` but may be higher
// if we are not network constrained.
/*
设置当前估计的网络属性。`target_bitrate` 为零时会暂停编码器。 `stable_target_bitrate`
是 `target_bitrate` 的一个过滤版本。它总是小于或等于 `target_bitrate`。它可以用来避免
昂贵编码设置（如分辨率）的快速变化。`link_allocation` 是网络链路上可用于此视频流的带宽。
它总是至少为 `target_bitrate`，但如果我们不受网络限制，可能会更高。
*/
virtual void OnBitrateUpdated(DataRate target_bitrate,
DataRate stable_target_bitrate,
DataRate link_allocation,
uint8_t fraction_lost,
int64_t round_trip_time_ms,
double cwnd_reduce_ratio) = 0;
// Set a FecControllerOverride, through which the encoder may override
// decisions made by FecController.
/*
设置一个 FecControllerOverride，通过它编码器可以覆盖 FecController 所做的决策
*/
virtual void SetFecControllerOverride(
FecControllerOverride* fec_controller_override) = 0;
// Creates and configures an encoder with the given `config`. The
// `max_data_payload_length` is used to support single NAL unit
// packetization for H.264.
/*
使用给定的 `config` 创建并配置一个编码器。`max_data_payload_length` 用于支持 H.264
的单个 NAL 单元打包。
*/
virtual void ConfigureEncoder(VideoEncoderConfig config,
size_t max_data_payload_length) = 0;
virtual void ConfigureEncoder(VideoEncoderConfig config,
size_t max_data_payload_length,
SetParametersCallback callback) = 0;
// Permanently stop encoding. After this method has returned, it is
// guaranteed that no encoded frames will be delivered to the sink.
virtual void Stop() = 0;
};
1.2 资源类（Resource）
上面关于Resource类的声明位于api/adaptation/resource.h中，这里的可以表示帧率，码率等编码信息，这个类存在的意义是对这些信息进行监控，如果某个资源超出了预期范围，需要将这个资源进行调控。
// A Resource monitors an implementation-specific resource. It may report
// kOveruse or kUnderuse when resource usage is high or low enough that we
// should perform some sort of mitigation to fulfil the resource's constraints.
//
// The methods on this interface are invoked on the adaptation task queue.
// Resource usage measurements may be performed on an any task queue.
//
// The Resource is reference counted to prevent use-after-free when posting
// between task queues. As such, the implementation MUST NOT make any
// assumptions about which task queue Resource is destructed on.
/*
1.一个资源监控特定实现的资源。当资源使用量足够高或足够低，以至于我们应该执行某种缓解
措施以满足资源的约束时，它可能会报告kOveruse（过载）或kUnderuse（欠载）。
2.这个接口上的方法在适应性任务队列上被调用。资源使用量的测量可以在任何任务队列上执行。
3.资源被引用计数，以防止在任务队列之间发布时出现使用后释放的问题。因此，
实现必须不对资源被销毁的任务队列做任何假设。
*/
class RTC_EXPORT Resource : public RefCountInterface {
public:
Resource();
// Destruction may happen on any task queue.
~Resource() override;
virtual std::string Name() const = 0;
// The `listener` may be informed of resource usage measurements on any task
// queue, but not after this method is invoked with the null argument.
virtual void SetResourceListener(ResourceListener* listener) = 0;
};
// ResourceListenter和ResourceUsageState的声明如下
enum class ResourceUsageState {
// Action is needed to minimze the load on this resource.
// 过载，需要降低这个资源
kOveruse,
// Increasing the load on this resource is desired, if possible.
// 欠载，需要增加这个资源
kUnderuse,
};
RTC_EXPORT const char* ResourceUsageStateToString(
ResourceUsageState usage_state);
class RTC_EXPORT ResourceListener {
public:
virtual ~ResourceListener();
// 评估当前资源的状态
virtual void OnResourceUsageStateMeasured(
rtc::scoped_refptr<Resource> resource,
ResourceUsageState usage_state) = 0;
};
1.3 已编码图像的回调（EncodedImageCallback）
EncodedImageCallback用于图像已经编码之后的回调，声明位于api/video_codecs/video_encoder.h中
class RTC_EXPORT EncodedImageCallback {
public:
virtual ~EncodedImageCallback() {}
struct Result {
enum Error {
OK,
// Failed to send the packet.
ERROR_SEND_FAILED,
};
explicit Result(Error error) : error(error) {}
Result(Error error, uint32_t frame_id) : error(error), frame_id(frame_id) {}
Error error;
// Frame ID assigned to the frame. The frame ID should be the same as the ID
// seen by the receiver for this frame. RTP timestamp of the frame is used
// as frame ID when RTP is used to send video. Must be used only when
// error=OK.
/*
分配给帧的帧ID。该帧ID应与接收方看到的该帧的ID相同。当使用RTP发送视频时，
帧的RTP时间戳被用作帧ID。仅当错误为OK时必须使用。
*/
uint32_t frame_id = 0;
// Tells the encoder that the next frame is should be dropped.
// 告诉编码器下一个帧应该被丢弃。
bool drop_next_frame = false;
};
// Used to signal the encoder about reason a frame is dropped.
// kDroppedByMediaOptimizations - dropped by MediaOptimizations (for rate
// limiting purposes).
// kDroppedByEncoder - dropped by encoder's internal rate limiter.
// TODO(bugs.webrtc.org/10164): Delete this enum? It duplicates the more
// general VideoStreamEncoderObserver::DropReason. Also,
// kDroppedByMediaOptimizations is not produced by any encoder, but by
// VideoStreamEncoder.
/*
用于向编码器发出信号，说明帧被丢弃的原因。
kDroppedByMediaOptimizations: 由 MediaOptimizations 丢弃（出于速率限制的目的）。
kDroppedByEncoder: 由编码器的内部速率限制器丢弃。
TODO(bugs.webrtc.org/10164): 删除这个枚举？它与更通用的 VideoStreamEncoderObserver::DropReason 重复。此外，
kDroppedByMediaOptimizations 不是由任何编码器产生的，而是由 VideoStreamEncoder 产生的。
*/
enum class DropReason : uint8_t {
kDroppedByMediaOptimizations,
kDroppedByEncoder
};
// Callback function which is called when an image has been encoded.
// 在一帧被编码之后会被调用
virtual Result OnEncodedImage(
const EncodedImage& encoded_image,
const CodecSpecificInfo* codec_specific_info) = 0;
// 丢弃一帧
virtual void OnDroppedFrame(DropReason /* reason */) {}
};
1.4 视频资源监听器（VideoSourceRestrictionsListener）
// The listener is responsible for carrying out the reconfiguration of the video
// source such that the VideoSourceRestrictions are fulfilled.
// 监听器负责执行视频源的重新配置，以满足 VideoSourceRestrictions 的要求。
class VideoSourceRestrictionsListener {
public:
virtual ~VideoSourceRestrictionsListener();
// The `restrictions` are filtered by degradation preference but not the
// `adaptation_counters`, which are currently only reported for legacy stats
// calculation purposes.
/*
`restrictions` 会根据降级偏好进行过滤，但 `adaptation_counters` 不会，
它们目前仅用于传统统计计算目的的报告。
*/
virtual void OnVideoSourceRestrictionsUpdated(
VideoSourceRestrictions restrictions,
const VideoAdaptationCounters& adaptation_counters,
rtc::scoped_refptr<Resource> reason,
const VideoSourceRestrictions& unfiltered_restrictions) = 0;
};
总体来讲，VideoStreamEncoder完成了进行视频编码的第一道工序，包括获取视频帧，根据编码情况进行码率调整，根据网络情况丢帧等等，由于涵盖内容太广泛，一些其他很重要的类没有分析，例如DegradationPreferenceManager、QpParser等，后续单独记录
