论文：Detecting and Correcting for Label Shift with Black Box Predictors（BBSE）

foreword

If you are interested in this article, you can click " 【Visitor Must Read-Guide Page】This article includes all high-quality blogs on the homepage " to view the complete blog classification and corresponding links.

---

overview

Let’s start with an example of influenza. In August, the hospital trained the model $f$ , assuming its feature$x$ is "with or without cough", and the predicted label$y$ is "whether you have flu or not".

Follow-up months model $f$ was working well, but by February of the following year, the hospital found that$f$ predicts that the number of people who "get the flu" has increased significantly. At this time, we know that this is related to "winter is a high incidence of flu". But a problem arose immediately, the fftrained with August data$Whether\; f$ can also be effectively predicted in February, and whether the prior learned from the data in August will affect the judgment in February.

Formalizing the problem, we can find that $p(and\mid \mathbf{x})=p($ flu∣$|cough$ )$)$和$p\left(and\right)=p($ flu$)$ has clearly changed, so previous studies on "covariate shift" are no longer applicable.

Going deeper, we can find that $p(\mathbf{x}\mid y)=p($ cough∣$|Flu$ )$)$ does not seem to have changed much, thus introducing the "label shift" problem that this article focuses on, which represents the following situation:

• label marginal distribution $p\left(y\right)$ changes, but the conditional distribution$p(x|y)$ unchanged

Then the paper proposes the "Black Box Shift Estimation (BBSE)" method, which uses the "black box predictor" to estimate the changing $p\left(y\right)$ , and only its corresponding "confusion matrices" are required to be invertible, even if the predictor is biased, inaccurate or uncalibrated.

---

problem setting

源域：$\mathcal{X}\times$Distribution PPon$Y$$P$，$D=\{({\mathbf{x}}_i,y_i){\}}_{i=1}^n$, based on $D$ trained black box model$f:X\to \mathcal{Y}$

目标域：$X\times$Distribution QQon$Y$$Q$，$X^{\prime }=[{\mathbf{x}}_1^{\prime };...;{\mathbf{x}}_m^{\prime }]$

目标: detect $P\to QWhether$ a "label shift" has occurred, and if so, retrain the model to adapt to the distribution$Q$

三大假设：

• 「label shift / target shift」假设：
  $$p(\mathbf{x}\mid y)=q(\mathbf{x}|y)\forall x\in \mathcal{X},y\in \mathcal{Y}$$
• $\forall y\in Y$，若$q(y)>0$则$p\left(and\right)>0$
• $The\; confusion\; matrix\; corresponding\; to\; f$ (confusion matrix)${\mathrm{C}}_p\left(f\right)$ is invertible, and the matrix is ​​defined as follows:
  $${\mathbf{C}}_P(f):=p(f(x),y)\in {\mathbb{R}}^{|\mathcal{Y}|\times |\mathcal{Y}|}$$

---

B.B.S.E.

The "Black Box Shift Estimation (BBSE)" method is mainly used to estimate $w(y)=q\left(y\right)/p\left(y\right)$ , also define the following:
$$\begin{array}{rl}q(\hat{y})& =\sum _{y\in \mathcal{Y}}q(\hat{y}\mid and)q\left(and\right)\\ & =\sum _{y\in Y}p(\hat{y}|and\left)q\right(and)=\sum _{y\in Y}p(\hat{y},y)\frac{q(y)}{p\left(and\right)}\end{array}$$

where $\hat{y}$ie ffThe pseudo-label given by$f$$q(\hat{y}\mid y)=p(\hat{y}|y)$则来自于下述推导：
$$\begin{array}{rl}& q(\hat{y}\mid y)=\sum _{y\in \mathcal{Y}}q(\hat{y}|\mathbf{x},y)q(\mathbf{x}\mid y)=\sum _{y\in \mathcal{Y}}q(\hat{y}|\mathbf{x},y)p(\mathbf{x}|y)\\ & =\sum _{y\in Y}{p}_f(\hat{y}|\mathbf{x})p(\mathbf{x}|y)=\sum _{y\in Y}p(\hat{y}|x,y\left)p\right(x|y)=p(\hat{y}|y)\end{array}$$

The key part is that $q(\mathbf{x}|y)=p(\mathbf{x}|y)$ Specifies the following equations:
$$\begin{array}{c}{m}_{\hat{y}}={\mathrm{C}}_{\hat{y}\mid y}{m}_y=C_{\hat{y},y}w\\ \hat{w}={\hat{C}}_{\hat{y},y}^{-1}{\hat{m}}_{\hat{y}}\\ {\hat{m}}_y=\mathrm{diag}\left({\hat{n}}_y\right)\hat{w}\end{array}$$

The symbols are defined as follows, and the core idea is the formula at the beginning of this section, but a large number of symbols are introduced for the sake of rigor, but the essence remains the same.

Theoretical guarantee

The first is the guarantee of "Consistency";

the second is the guarantee of "Error bounds":

According to the results of "Error bounds" above, it can be found that when choosing a black-box model, " ${\mathrm{C}}_{\hat{y},y}$The larger the minimum singular value, the more suitable the model is.

---

Label-Shift Detection

Under the first three assumptions, $q(y)=p\left(and\right)\iff p(\hat{y})=q(\hat{y})$ , so using "two-sample tests" for$p\left(\hat{y}\right)=q\left(\hat{y}\right)$ for detection.

---

Fit the model to the new distribution

Calculate $\stackrel{After\; ^}{\mathbf{w}}$ , use "importance weighted ERM" in the source domain datasetD \mathcal{D}Just retrain the model on$D$
$$\mathcal{L}=\sum _{i=1}^n{\hat{w}}_i\cdot \ell \left(y_i,{\mathbf{x}}_i\right)$$

The overall algorithm is as follows:

---

Detect Label-Shift hypothesis is true

Use "kernel two-sample tests" to test whether the following formula is true:
$${\mathbb{E}}_p\left[w\right(y\left)k\right(\varphi \left(x\right),\cdot )]=E_q\left[k\right(\varphi \left(x\right),\cdot )]$$

That is, it is transformed into the calculation of the following MMD distance:
$${\Vert \frac{1}{n}\sum _{i=1}^n\left[\hat{\mathbf{w}}\left(y_i\right)k\left(\varphi \left({\mathbf{x}}_i\right),\cdot \right)\right]-\frac{1}{m}\sum _{j=1}^{m}k\left(\varphi \left(x_j^{\prime }\right),\cdot \right)\Vert }_{\mathcal{H}}^2$$

---

References

• [ICML18 - Zachary C. Lipton] Detecting and Correcting for Label Shift with Black Box Predictors