【WebRTC-13】是在哪，什么时候，创建编解码器？

Android-RTC系列软重启，改变以往细读源代码的方式 改为 带上实际问题分析代码。增加实用性，方便形成肌肉记忆。同时不分种类、不分难易程度，在线征集问题切入点。

问题：编解码器的关键实体类是什么？在哪里&什么时候创建的？

这个问题是在分析webrtc如何增加第三方外置的编解码库时 额外提出来的，在找答案的过程中领略webrtc内部代码结构组织的划分。废话不多，这个问题的关键可以想到之前的一个问题  webrtc是如何确定双端的编解码类型？  是在sdp交换信息后，local和remote的description两者结合确认。那么可以在这基础上继续寻找，也就是去SdpOfferAnswerHandler找答案。

我们直接定位到 SdpOfferAnswerHandler:: ApplyLocalDescription / ApplyRemoteDescription。

RTCError SdpOfferAnswerHandler::ApplyLocalDescription(std::unique_ptr<SessionDescriptionInterface> desc,const std::map<std::string, const cricket::ContentGroup*>&  bundle_groups_by_mid) 
{pc_->ClearStatsCache();RTCError error = PushdownTransportDescription(cricket::CS_LOCAL, type);if (IsUnifiedPlan()) {UpdateTransceiversAndDataChannels(...)} else {... ...}UpdateSessionState(type, cricket::CS_LOCAL,local_description()->description(),bundle_groups_by_mid);// Now that we have a local description, we can push down remote candidates.UseCandidatesInRemoteDescription();... ...
}

大致的逻辑如上，这里关注 UpdateSessionState，继续深入。

RTCError SdpOfferAnswerHandler::UpdateSessionState(SdpType type,  cricket::ContentSource source,const cricket::SessionDescription* description,const std::map<std::string, const cricket::ContentGroup*>&  bundle_groups_by_mid) {// If this is answer-ish we're ready to let media flow.if (type == SdpType::kPrAnswer || type == SdpType::kAnswer) {EnableSending();}// Update the signaling state according to the specified state machine (see// https://w3c.github.io/webrtc-pc/#rtcsignalingstate-enum).if (type == SdpType::kOffer) {ChangeSignalingState(source == cricket::CS_LOCAL? PeerConnectionInterface::kHaveLocalOffer: PeerConnectionInterface::kHaveRemoteOffer);} else if (type == SdpType::kPrAnswer) {ChangeSignalingState(source == cricket::CS_LOCAL? PeerConnectionInterface::kHaveLocalPrAnswer: PeerConnectionInterface::kHaveRemotePrAnswer);} else {RTC_DCHECK(type == SdpType::kAnswer);ChangeSignalingState(PeerConnectionInterface::kStable);if (ConfiguredForMedia()) {transceivers()->DiscardStableStates();}}// Update internal objects according to the session description's media descriptions.return PushdownMediaDescription(type, source, bundle_groups_by_mid);
}

根据输入的type改变信令状态。注意最后的 PushdownMediaDescription，这里看函数名字有点奇怪，其核心功能是检索新的sdp信息，更新 rtp_transceivers 的channel

RTCError SdpOfferAnswerHandler::PushdownMediaDescription(SdpType type,  cricket::ContentSource source,const std::map<std::string, const cricket::ContentGroup*>&  bundle_groups_by_mid) 
{const SessionDescriptionInterface* sdesc =(source == cricket::CS_LOCAL ? local_description() : remote_description());// Push down the new SDP media section for each audio/video transceiver.auto rtp_transceivers = transceivers()->ListInternal();std::vector<std::pair<cricket::ChannelInterface*, const MediaContentDescription*>> channels;for (const auto& transceiver : rtp_transceivers) {const ContentInfo* content_info =FindMediaSectionForTransceiver(transceiver, sdesc);cricket::ChannelInterface* channel = transceiver->channel();const MediaContentDescription* content_desc = content_info->media_description();channels.push_back(std::make_pair(channel, content_desc));}for (const auto& entry : channels) {std::string error;bool success = context_->worker_thread()->BlockingCall([&]() {return (source == cricket::CS_LOCAL)? entry.first->SetLocalContent(entry.second, type, error): entry.first->SetRemoteContent(entry.second, type, error);});}return RTCError::OK();
}

（这里的channel以后介绍）伪代码如上。可以看到关于ChannelInterface的关键方法 SetLocalContent 、SetRemoteContent。

文件位置  src/pc/channel.cc
bool BaseChannel::SetLocalContent(const MediaContentDescription* content,SdpType type, std::string& error_desc) {RTC_DCHECK_RUN_ON(worker_thread());TRACE_EVENT0("webrtc", "BaseChannel::SetLocalContent");return SetLocalContent_w(content, type, error_desc);
}
bool BaseChannel::SetRemoteContent(const MediaContentDescription* content,SdpType type, std::string& error_desc) {RTC_DCHECK_RUN_ON(worker_thread());TRACE_EVENT0("webrtc", "BaseChannel::SetRemoteContent");return SetRemoteContent_w(content, type, error_desc);
}

SetLocalContent_w / SetRemoteContent_w又到具体的媒体通道类VoiceChannel / VideoChannel实现，以VideoChannel为例，精简核心代码如下。

bool VideoChannel::SetLocalContent_w(const MediaContentDescription* content,SdpType type, std::string& error_desc) 
{RtpHeaderExtensions header_extensions =GetDeduplicatedRtpHeaderExtensions(content->rtp_header_extensions());media_send_channel()->SetExtmapAllowMixed(content->extmap_allow_mixed());VideoReceiverParameters recv_params = last_recv_params_;VideoSenderParameters send_params = last_send_params_;MediaChannelParametersFromMediaDescription(content, header_extensions,webrtc::RtpTransceiverDirectionHasRecv(content->direction()),&recv_params);media_receive_channel()->SetReceiverParameters(recv_params);media_send_channel()->SetSenderParameters(send_params);UpdateLocalStreams_w(content->streams(), type, error_desc)UpdateMediaSendRecvState_w();
}

这里的UpdateLocalStreams_w又回到了BaseChannel。这里有一大段注释比较关键，主要描述了：在媒体协商的过程中SSRC与StreamParams相关联，构成安全的 local_stream_成员对象。

bool BaseChannel::UpdateLocalStreams_w(const std::vector<StreamParams>& streams,SdpType type, std::string& error_desc) 
{// In the case of RIDs (where SSRCs are not negotiated), this method will// generate an SSRC for each layer in StreamParams. That representation will// be stored internally in `local_streams_`.// In subsequent offers, the same stream can appear in `streams` again// (without the SSRCs), so it should be looked up using RIDs (if available)// and then by primary SSRC.// In both scenarios, it is safe to assume that the media channel will be// created with a StreamParams object with SSRCs. However, it is not safe to// assume that `local_streams_` will always have SSRCs as there are scenarios// in which niether SSRCs or RIDs are negotiated.... ...media_send_channel()->AddSendStream(new_stream);
}

到这里我们先停顿一下，因为发现这里出现了众多 Channel 对象，ChannelInterface、BaseChannel、VoiceChannel/VideoChannel、media_send_channel/media_receive_channel。它们究竟是什么关系？我绘制了一张简易的UML图，这张图概括了Channel 以及之后要介绍的 Stream的内部关系，提取了核心代码的常见方法。大家一定要放大仔细看看！

小结：这部分阐述了 webrtc如何从sdp提取信息，根据ssrc创建并绑定网络传输通器中的 Channel。并通过代码，由BaseChannel->Video/VoiceChannel承接rtp数据包。

---

有了上面的UML图预热，在进入 media_send_channel()->AddSendStream的流程之前，我们要搞清楚这个 media_send_channel是怎么来的。

回到文章最开始的 SdpOfferAnswerHandler::ApplyLocalDescription 的UpdateTransceiversAndDataChannels。关键代码逻辑如下，可以看到涉及ChannelInterface的通道，是由RtpTransceiver内部创建的。

RTCError SdpOfferAnswerHandler::UpdateTransceiversAndDataChannels(cricket::ContentSource source,const SessionDescriptionInterface& new_session,const SessionDescriptionInterface* old_local_description,const SessionDescriptionInterface* old_remote_description,const std::map<std::string, const cricket::ContentGroup*>& bundle_groups_by_mid) 
{const ContentInfos& new_contents = new_session.description()->contents();for (size_t i = 0; i < new_contents.size(); ++i) {const cricket::ContentInfo& new_content = new_contents[i];auto transceiver_or_error =AssociateTransceiver(source, new_session.GetType(), i, new_content,old_local_content, old_remote_content);auto transceiver = transceiver_or_error.Value();RTCError error= UpdateTransceiverChannel(transceiver, new_content, bundle_group);}
}RTCError SdpOfferAnswerHandler::UpdateTransceiverChannel(rtc::scoped_refptr<RtpTransceiverProxyWithInternal<RtpTransceiver>> transceiver,const cricket::ContentInfo& content,const cricket::ContentGroup* bundle_group) 
{cricket::ChannelInterface* channel = transceiver->internal()->channel();if (!channel) {auto error = transceiver->internal()->CreateChannel(...);}
}

RtpTransceiver的CreateChannel内部，其核心是调用media_engine去创建对应的SendChannel / ReceiveChannel，最终组成RtpTransceiver的 VideoChannel / VoiceChannel。

RTCError RtpTransceiver::CreateChannel(absl::string_view mid,Call* call_ptr,const cricket::MediaConfig& media_config,bool srtp_required,CryptoOptions crypto_options,const cricket::AudioOptions& audio_options,const cricket::VideoOptions& video_options,VideoBitrateAllocatorFactory* video_bitrate_allocator_factory,std::function<RtpTransportInternal*(absl::string_view)> transport_lookup) 
{std::unique_ptr<cricket::ChannelInterface> new_channel;if (media_type() == cricket::MEDIA_TYPE_VIDEO) {std::unique_ptr<cricket::VideoMediaSendChannelInterface>media_send_channel = media_engine()->video().CreateSendChannel(...);std::unique_ptr<cricket::VideoMediaReceiveChannelInterface>media_receive_channel = media_engine()->video().CreateReceiveChannel(...);new_channel = std::make_unique<cricket::VideoChannel>(worker_thread(), network_thread(), signaling_thread(), std::move(media_send_channel), std::move(media_receive_channel), ...);} else {// media_type() == cricket::MEDIA_TYPE_AUDIO}SetChannel(std::move(new_channel), transport_lookup);return RTCError::OK();
}

根据以往的文章，我们就可以快速定位到 src/media/engine/webrtc_video_engine / webrtc_audio_engine，找到SendChannel ReceiveChannel。至此，我们正式定位到了media_send_channel()的具体实现。

// 以media_type==video为例
std::unique_ptr<VideoMediaSendChannelInterface>
WebRtcVideoEngine::CreateSendChannel(webrtc::Call* call,const MediaConfig& config,const VideoOptions& options,const webrtc::CryptoOptions& crypto_options,webrtc::VideoBitrateAllocatorFactory* video_bitrate_allocator_factory) {return std::make_unique<WebRtcVideoSendChannel>(call, config, options, crypto_options, encoder_factory_.get(),decoder_factory_.get(), video_bitrate_allocator_factory);
}
std::unique_ptr<VideoMediaReceiveChannelInterface>
WebRtcVideoEngine::CreateReceiveChannel(webrtc::Call* call,const MediaConfig& config,const VideoOptions& options,const webrtc::CryptoOptions& crypto_options) {return std::make_unique<WebRtcVideoReceiveChannel>(call, config, options, crypto_options, decoder_factory_.get());
}

小结：根据m=session创建RtpTransceiver，Video/VoiceChannel由webrtc_medie_engine创建，并保存在RtpTransceiver网络传器者的成员变量。Video/VoiceChannel 内包含SendChannel和ReceiveChannel。

---

回头再看media_send_channel()->AddSendStream(new_stream)，即WebRtcVideoSendChannel::AddSendStream，其核心逻辑很简单：

bool WebRtcVideoSendChannel::AddSendStream(const StreamParams& sp) 
{WebRtcVideoSendStream* stream = new WebRtcVideoSendStream(call_, sp, std::move(config), default_send_options_,video_config_.enable_cpu_adaptation, bitrate_config_.max_bitrate_bps,send_codec(), send_rtp_extensions_, send_params_);uint32_t ssrc = sp.first_ssrc();send_streams_[ssrc] = stream;
}

WebRtcVideoSendStream 的构造函数内容比较多，但都是属性赋值。我们这里只关心文章提出的问题，也就是构造函数里唯一调用的成员函数SetCodec。

// src/media/engine/webrtc_video_engine.cc
void WebRtcVideoSendChannel::WebRtcVideoSendStream::SetCodec(const VideoCodecSettings& codec_settings) 
{parameters_.encoder_config = CreateVideoEncoderConfig(codec_settings.codec);parameters_.config.rtp = ...parameters_.codec_settings = codec_settings;// TODO(bugs.webrtc.org/8830): Avoid recreation, it should be enough to call  ReconfigureEncoder.RTC_LOG(LS_INFO) << "RecreateWebRtcStream (send) because of SetCodec.";RecreateWebRtcStream();
}void WebRtcVideoSendChannel::WebRtcVideoSendStream::RecreateWebRtcStream() 
{if (stream_ != NULL) {call_->DestroyVideoSendStream(stream_);}stream_ = call_->CreateVideoSendStream(std::move(config),parameters_.encoder_config.Copy());// Attach the source after starting the send stream to prevent frames from// being injected into a not-yet initializated video stream encoder.// rtc::VideoSourceInterface<webrtc::VideoFrame>* source_if (source_) {stream_->SetSource(source_, GetDegradationPreference());}
}

具体实现还要到Call::CreateVideoSendStream。这里有个细节，stream_->SetSource(webrtc::VideoFrame )出现了VideoFrame，显然路是找对了，继续往下。

//  src/call/call.cc
webrtc::VideoSendStream* Call::CreateVideoSendStream(webrtc::VideoSendStream::Config config,VideoEncoderConfig encoder_config,std::unique_ptr<FecController> fec_controller) 
{VideoSendStreamImpl* send_stream = new VideoSendStreamImpl(...);for (uint32_t ssrc : ssrcs) {RTC_DCHECK(video_send_ssrcs_.find(ssrc) == video_send_ssrcs_.end());video_send_ssrcs_[ssrc] = send_stream;}video_send_streams_.insert(send_stream);video_send_streams_empty_.store(false, std::memory_order_relaxed);
}// src/video/video_send_stream_impl.cc
VideoSendStreamImpl::VideoSendStreamImpl(RtcEventLog* event_log,VideoSendStream::Config config,VideoEncoderConfig encoder_config,std::unique_ptr<FecController> fec_controller,const FieldTrialsView& field_trials,std::unique_ptr<VideoStreamEncoderInterface> video_stream_encoder_for_test)
//构造赋值
: video_stream_encoder_(video_stream_encoder_for_test? std::move(video_stream_encoder_for_test): CreateVideoStreamEncoder(...) ), ... ...
//构造VideoStreamEncoder
std::unique_ptr<VideoStreamEncoderInterface> CreateVideoStreamEncoder() {std::unique_ptr<TaskQueueBase, TaskQueueDeleter> encoder_queue =task_queue_factory->CreateTaskQueue("EncoderQueue",TaskQueueFactory::Priority::NORMAL);TaskQueueBase* encoder_queue_ptr = encoder_queue.get();return std::make_unique<VideoStreamEncoder>(...std::move(encoder_queue), ...);
}

到这里我们就找到了webrtc的视频编码实体类VideoStreamEncoder（src/video/video_stream_encoder.cc），相对应的解码实体类VideoStreamDecoder。这里放一些关键方法，此类是个宝库，任何关于视频编码功能的细节，都可以在这找到参考。

---

总结答案：整篇文章跟踪的代码逻辑如下，归纳了从Sdp->RtpTransceiver->VideoChannel/VoiceChannel->Send&ReceiveChannel-> 然后根据sdp::ssrc创建VideoSendStream-> VideoStreamEncoder。

还待挖掘的细节非常多，奈何篇幅有限有所侧重。在写本文章的时候，其实是在研究icecandidate，stun服务端已经搭建起来了。希望有兴趣的同学联系我，一起深入成长。

SdpOfferAnswerHandler:: ApplyLocalDescription / ApplyRemoteDescription（sdp信息）
SdpOfferAnswerHandler::UpdateTransceiversAndDataChannels -> UpdateTransceiverChannel（创建RtpTransceiver->Video/VoiceChannel）
SdpOfferAnswerHandler::UpdateSessionState
SdpOfferAnswerHandler::PushdownMediaDescription
BaseChannel::SetLocalContent(const MediaContentDescription* content, ..)
VoiceChannel/VideoChannel::SetLocalContent_w
BaseChannel::UpdateLocalStreams_w(const std::vector<StreamParams>& streams, ..)WebRtcVideoSendChannel::AddSendStream
WebRtcVideoSendChannel::WebRtcVideoSendStream::WebRtcVideoSendStream（Constructor）
WebRtcVideoSendChannel::WebRtcVideoSendStream::SetCodec|::RecreateWebRtcStream|::SetSenderParameters|::ReconfigureEncoder
Call::CreateVideoSendStream
VideoSendStreamImpl() -> VideoStreamEncoder(Interface)