【论文阅读】Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

论文下载
GitHub
bib:

@INPROCEEDINGS{,title		= {Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning},author	= {Eric Arazo and Diego Ortego and Paul Albert and Noel E O'Connor and Kevin McGuinness},booktitle	= {IJCNN},year		= {2020},pages     = {1--8}
}

1. 摘要

Semi-supervised learning, i.e. jointly learning from labeled and unlabeled samples, is an active research topic due to its key role on relaxing human supervision.

总览半监督学习。

In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples.

提到半监督分类中的一致性正则。

We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions.

提到本文中适用了伪标签技术（soft pseudo-labels）。

We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it.

核心的贡献。提出了确认偏差（confirmation bias），本文贡献是证明了mixup augmentation和setting a minimum number of labeled samples per mini-batch是有效减少确认偏差的正则技术。

The proposed approach achieves state-of-the-art results in CIFAR-10/100, SVHN, and Mini-ImageNet despite being much simpler than other methods.

These results demonstrate that pseudo-labeling alone can outperform consistency regularization methods, while the opposite was supposed in previous work.

这一点就很令人惊讶了，伪标签技术的方法超过了一致性正则的方法。还没看原文，应该是还没有出现FixMatch和FlexMatch方法。

2. 算法描述

符号	意义
$D_l=\{(x_i,y_i){\}}_{i=1}^{N_l}$	有标记数据
D u = { x i } i = 1 N u D_u = \{x_i\}^{N_u}_{i=1} Du​={xi​}i=1Nu​​	无标记数据
${\tilde{D}}_u=\{(x_i,{\tilde{y}}_i{\}}_{i=1}^N$	训练数据，其中对于有标记数据${\tilde{y}}_i$表示真实标签，对于无标记数据${\tilde{y}}_i$表示对应伪标签。
$h_{\theta }$	模型及对应的参数$\theta$

经典的交叉熵损失函数:
$${\ell }^{\ast }(\theta )=-\sum _{i=1}^N{\tilde{y}}_i^{\mathsf{T}}\log (h_{\theta }(x_i))\text{\hspace{1em}(1)}$$
Note:

In particular, we store the softmax predictions $h_{\theta }(x_i)$ of the network in every mini-batch of an epoch and use them to modify the soft pseudo-label $\tilde{y}$ for the $N_u$ unlabeled samples at the end of every epoch.

We proceed as described from the second to the last training epoch, while in the first epoch we use the softmax predictions for the unlabeled samples from a model trained in a 10 epochs warm-up phase using the labeled data subset $D_u$.

Soft pseudo-labels在本文中表示上一个阶段网络对于无标记样本的预测。注意区别于Hard pseudo-labels，Soft pseudo-labels不是one-hot向量，而是对于样本预测的概率向量（softmax）。

Two Regularizations:
$$\ell ={\ell }^{\ast }+{\lambda }_A{R}_A+{\lambda }_H{R}_H\text{\hspace{1em}(2)}$$
where

• $R_A={\sum }_{c=1}^C{p}_c\log (\frac{p_c}{{\overline{h}}_c})$;
• $R_H=-\frac{1}{N}{\sum }_{i=1}^N{\sum }_{c=1}^C{h}_{\theta }^c(x_i)\log (h_{\theta }^c(x_i))$.

$R_A$不鼓励将所有样本分配到单个类。其中$p_c$表示类别$c$的先验概率分布， ${\overline{h}}_c$表示模型在数据集中所有$c$类别样本中的平均概率（softmax）。意思是本来有猫有狗的类别，网络为了省事，直接不管三七二十一，直接预测一个猫，这个现象在不平衡数据集上很容易出现。

$R_H$（entropy regularization）鼓励每个软伪标记的概率分布集中在单个类上，避免了网络可能因弱引导而陷入的局部最优。这一点容易理解，就是对于一个样本，鼓励预测的类的概率远远大于其他类别。

Confirmation bias:

Overfitting to incorrect pseudo-labels predicted by the network is known as confirmation bias.
It is natural to think that reducing the confidence of the network on its predictions might alleviate this problem and improve generalization.

Note: 这里将确认偏差（confirmation bias）定义为网络对于不正确伪标签的过拟合。降低对于不正确标签的权重可以缓解这一现象。

mixup regularization:

Recently, mixup data augmentation introduced a strong regularization technique that combines data augmentation with label smoothing, which makes it potentially useful to deal with this bias.

Question:

• mixup的细节，在单个批次中，怎么mixup？
• mixup样本的标签如何确定？

setting a minimum number of labeled samples per mini-batch:

Oversampling the labelled examples by setting a minimum number of labeled samples per mini-batch k (as done in other works provides a constant reinforcement with correct labels during training, reducing confirmation bias and helping to produce better pseudo-labels.

Question:

• 单个批次样本如何配置，多少个有标记数据，多少个无标记数据？