机器学习与高维信息检索 - Note 3 - 逻辑回归（Logistic Regression）及相关实例

3. 逻辑回归

在谈论逻辑回归时，一般的设定是，我们有数据点$X\in {\mathbb{R}}^p$和输出变量 Y ∈ { − 1 , + 1 } Y\in\{-1,+1\} Y∈{ −1,+1}。这是一个所谓的二元分类问题。

其任务是在预定的函数类别$\mathcal{F}$中找到函数$f$，使$\mathrm{sign}f\left(X\right)$ predicts YYas well as possible$Y.$ _ A commonly used loss function to measure the "accuracy" of a prediction function is motivated by the number of misclassifications, i.e. if$The\; sign\; of\; f\left(X\right)$ and the real output$The\; sign\; of\; Y$ is inconsistent. We use a 0.1 loss function

$$L_{0,1}(Y,f(X))=\{\begin{array}{ll}1& \text{ if }Y\mathrm{sign}f(X)\le 0\\ 0& \text{ otherwise }\end{array}\text{\hspace{1em}(3. 1)}$$

来完成这项工作。

为了在给定的训练样本集${\left(x_i,y_i\right)}_{i=1,\dots ,n}$中找到最佳预测函数$f\in \mathcal{F}$，目标是找到最小化经验预期损失的$f\in \mathcal{F}$。

$$\frac{1}{n}\sum _{i=1}^n{L}_{0,1}\left(y_i,f\left(x_i\right)\right)\text{\hspace{1em}(3.2)}$$

However, finding the minimum for this problem is numerically infeasible. Even if we only consider a class of affine functions ${\mathcal{F}}_{\text{aff }}$, which is also difficult to solve numerically because the loss function is neither continuous nor separable. To make it easier to solve, we use a convex loss function to approximate the (non-continuous, non-convex) function $L_{0,1}$。

Supplement: Convexity

Definition 3.1

令$\mathcal{C}\subset {\mathbb{R}}^n$ is a convex set, that is, for any pair of elements$x_1,x_2\in For\; C$ , point$t{\mathbf{x}}_2+(1-t){\mathbf{x}}_1$Also C \mathcal{C}An element of$C$$t\in [0,1]$。如果对于所有${\mathbf{x}}_1,{\mathbf{x}}_2\in \mathcal{C},t\in [0,1]$，有$tf\left({\mathbf{x}}_2\right)+(1-t)f\left({\mathbf{x}}_1\right)\ge f\left(t{\mathbf{x}}_2+(1-t){\mathbf{x}}_1\right)$，一个函数$f:\mathcal{C}\to \mathbb{R}$被称为凸的。如果不等式是严格的，它就被称为严格凸的。

例: $f:{\mathbb{R}}^{+}\to \mathbb{R},x\mapsto 1/x$是凸的。

定理3.2

如果$f，g$是凸的，那么

• $h=\max (f,g)$
• $h=f+g$
• $h=g\circ f$ if $g$ is non-decreasing

图3.1：$L_{0,1}$和对数损失，横轴为$t:=yf(x)$。

定理3.3

严格凸函数的局部最小值与它的全局最小值相吻合。如果它存在的话，它是唯一的。

对于目前的问题，这意味着如果我们选择$f$是仿射的（即$f(\mathbf{x})=$ ${\mathbf{w}}^{\top }\mathbf{x}+b$，根据定义是凸的），作为损失函数，我们使用凸的对数损失函数$\ell (t)=log\left(1+e^{-t}\right)$。两者的结合是凸的。为了看到这一点，计算二阶导数并研究其黑森矩阵。结果发现，它只有非负的特征值。选择对数损失的原因是，它可以被解释为$L_{0,1}$损失的凸近似值，如图3.1所示。给定一些训练数据 { ( x i , y i ) } i = 1 , … , N \left\{\left(\mathbf{x}_{i}, y_{i}\right)\right\}_{i=1, \ldots, N} { (xi​,yi​)}i=1,…,N​, we can find the optimal parameters ww by solving the optimization problem$w$ and$b$。

$$\underset{\mathbf{w}\in {\mathbb{R}}^p,b\in \mathbb{R}}{\min }\frac{1}{n}\sum _{i=1}^n\log \left(1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)\right).\text{\hspace{1em}(3. 3)}$$

Convexity of cost function

For simplicity, we only consider linear ffcost function of$f\; .$to affine$The\; expansion\; of\; f$ is straightforward. Let us express it by
$$F(\mathbf{w})=\sum _{i=1}^n\log \left(1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{x}_i\right)\right)\right).$$

We also use the auxiliary function $g(z)=1/\left(1+e^{-z}\right)$，其中$g^{\prime }(z)=g(z)(1-g(z\left)\right)$ . So,$F$的一次和二次偏导是
$$\frac{\partial }{\partial w^{(j)}}F(\mathbf{w})=-\sum _i{y}_i{x}_i^{(j)}\left(1-g\left(y_i{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)$$

and

$$\frac{\partial }{\partial w^{(j)}\partial w^{(k)}}F(\mathbf{w})=\sum _i{y}_i^2{x}_i^{\left(j\right)}{x}_i^{(k)}g\left(y_i{w}^{\top }{x}_i\right)\left(1-g\left(y_i{w}^{\top }{x}_i\right)\right),$$

Among them $y_i^2=1$。为了证明该函数为非负定值，我们需要证明对所有$a$而言$a^{\top }{\nabla }^{2}Fa\ge 0$。我们定义辅助变量$P_i=g\left(y_i{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\left(1-g\left(y_i{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)$和${\rho }_i^{(j)}=x_i^{(j)}\sqrt{P_i}$。那么有

$$a^{\top }{\nabla }^{2}Fa=\sum _i\sum _{j,k}{a}_i{a}_j{x}_i^{(j)}{x}_i^{(k)}{P}_i=\sum _i{a}^{\top }{\rho }_i{r}_i^{\top }a\ge 0.$$

This is true for any combination of convex and affine functions.

In the following, we provide a probabilistic interpretation of this optimization problem. First, note that given the observed $Y=$Conditional probability x of$y$$x$ is defined as

$$Pr(Y=y\mid x)=\exp (-\ell (y,f(x)))=\frac{1}{1+\exp \left(-y\left({\mathbf{w}}^{\top }\mathbf{x}+b\right)\right)}.\text{\hspace{1em}(3. 4)}$$

为了找到（3.3）的解决方案，通常的做法是使用基于梯度的方法。最简单的形式是梯度下降法，在一个给定的迭代点，我们计算梯度，然后在该梯度的负方向迈出一步。函数 $F(\mathbf{w},b)={\sum }_i\log \left(1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)\right)$可以通过计算偏导来确定

$$\begin{array}{rl}\frac{\partial }{\partial w^{(k)}}F(\mathbf{w},b)& =\sum _{i=1}^n\frac{\exp \left(-y_i\left(w^{\top }{x}_i+b\right)\right)}{1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)}\left(-y_i{x}_i^{(k)}\right)\\ & =\sum _{i=1}^n\frac{1}{1+\exp \left(y_i\left(w^{\top }{x}_i+b\right)\right)}\left(-y_i{x}_i^{\left(k\right)}\right)\\ & =\sum _{i\mid y_i=1}\frac{1}{1+\exp \left(w^{\top }{x}_i+b\right)}\left(-x_i^{\left(k\right)}\right)+\sum _{i|y_i=-1}\frac{1}{1+\exp \left(-{\mathbf{w}}^{\top }{x}_i+b\right)}\left(x_i^{\left(k\right)}\right)\end{array}$$

系数$1/\left(1+\exp \left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)$和 $1/\left(1+\exp \left(-{\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)$是错误的预测概率。(参照（3.4））

因此，当我们朝着负梯度的方向迈出一步时，我们就会朝着这些错误的 "反方向 "前进。这就是为什么梯度下降方法也被称为错误驱动方法的原因。当前模型的错误（这里由一些权重$(\mathbf{w},b)）$被用来改进它。梯度指向的方向是使当前模型中的错误最小化。

总的来说：

Logistic regression is a supervised classification method, the decision function is affine, and the loss is $L(y,f(\mathbf{x}))=\log \left(1+e^{-yf\left(x\right)}\right))$measure. Optimal parameters${\mathbf{w}}^{\star },b^{\star }$ is found by minimizing the empirical expected loss, that is,${\min }_{\mathbf{w},b}\frac{1}{N}\sum \log \left(1+e^{-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)}\right)$。一旦确定了最佳的${\mathbf{w}}^{\star },b^{\star }$，我们就可以通过计算$\mathrm{sign}\left({\mathbf{w}}^{\star \top }{\mathbf{x}}_{\text{new }}+b^{\star }\right)$对一个新数据点${\mathbf{w}}^{\star },b^{\star }$进行分类。我们还可以通过公式(3.4)计算出这种分类正确的概率。

3.1逻辑回归的替代方法

首先，请注意，逻辑回归这个名字可能会产生误导，因为事实上逻辑回归并不是一种回归方法，而是一种分类方法。上一节更多的是以优化为目的，而在这里，我们提供了一种更多的统计方法来处理逻辑回归。

例:
我们试图预测在某些参数下的死亡概率。让$x_1$为一个人的年龄，$x_2$为性别（0对应男性，1对应女性），$x_3$为胆固醇水平。我们假设可以将这些数值以线性方式组合起来，从而得到一个与死亡概率有某种关联的实际数值

$$w_0+w_1{x}_1+w_2{x}_2+w_3{x}_3={\mathbf{w}}^{\top }\mathbf{x}$$

有$\mathbf{x}={\left[1,x_1,x_2,x_3\right]}^{\top },\mathbf{w}={\left[w_0,w_1,w_2,w_3\right]}^{\top }$。$w_i$的值称为权重，$w_0$称为偏置。得到的值在$\mathbb{R}$中。为了将其塑造成一个概率，我们需要一个函数$\sigma$，将这个值压缩到$[0,1]$的区间内。一个能实现这一目的的函数是逻辑回归函数

$$\sigma (a)=\frac{1}{1+e^{-a}}\text{\hspace{1em}(3.5)}$$

得到的模型是$P(\text{死亡}\mid x)=\sigma \left({\mathbf{w}}^{\top }\mathbf{x}\right)$。

---

更一般地说，我们考虑二元分类问题的训练数据 D = { ( x 1 , z 1 ) , … , ( x n , z n ) } , x i ∈ R d , z i ∈ { 0 , 1 } D=\left\{\left(\mathbf{x}_{1}, z_{1}\right), \ldots,\left(\mathbf{x}_{n}, z_{n}\right)\right\}, \mathbf{x}_{i} \in \mathbb{R}^{d}, z_{i} \in\{0,1\} D={ (x1​,z1​),…,(xn​,zn​)},xi​∈Rd,zi​∈{ 0,1} ，输入和输出变量的依赖性模型为$z_i\propto \mathrm{Bernoulli}\left(\sigma \left({\mathbf{w}}^{\top }x\right)\right)$, where we assume$z_i$is independent.

In order to train this model, we need to find the given $D\text{'}$ swwThe maximum likelihood estimate of$w\; ,\; that\; is$

$${\mathbf{w}}_{\mathrm{M}\mathrm{L}\mathrm{E}}=\arg \underset{w}{\max }Pr(D\mid \mathbf{w})$$

have

$$Pr(D\mid \mathbf{w})=\prod _{i=1}^n\mathrm{Pr}\left(z_i|{\mathbf{x}}_i,w\right)=\prod _{i=1}^{n}p{\left({\mathbf{w}}^{\top }{\mathbf{x}}_i\right)}^{z_i}{\left(1-p\left(w^{\top }{x}_i\right)\right)}^{1-z_i}\text{\hspace{1em}(3.6)}$$

For optimization purposes, it is common to use the negative log ⁡ \log of the above conditional probability$\log$，即
$$L(\mathbf{w})=-\log Pr(D\mid w)=-\sum _{i=1}^n{z}_i\log \left(p\left({\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)+\left(1-z_i\right)\log \left(1-p\left(w^{\top }{x}_i\right)\right)\text{. }\text{\hspace{1em}(3.7)}$$

respective gradients (relative to $w$ ) is given by

$$\mathbf{g}={\nabla }_{\mathbf{w}}L(\mathbf{w})=\sum _{i=1}^n\left(p\left({\mathbf{w}}^{\top }{\mathbf{x}}_i\right)-z_i\right){\mathbf{x}}_i=\mathbf{X}\left(p\left({\mathbf{X}}^{\top }\mathbf{w}\right)-\mathbf{z}\right)\text{\hspace{1em}(3.8)}$$

与$\mathbf{X}=\left[{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\right]\in {\mathbb{R}}^{d\times n}$ . CorrespondingwwHessian matrixH \mathbf{H}$of\; w$$H$ is given as

$$\mathbf{H}={\nabla }_{\mathbf{w}}^{2}L(\mathbf{w})={XBX}^{\top }\text{\hspace{1em}(3.9)}$$

有$\mathbf{B}=\mathrm{diag}\left(\sigma \left({\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\left(1-\sigma \left({\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)\right)\in {\mathbb{R}}^{n\times n}$ 。黑塞矩阵$\mathbf{H}$是半正定的，因此$L$是凸的。

假设Hessian是可逆的。那么牛顿方法的迭代形式为

$$\begin{array}{rl}{\mathbf{w}}_{t+1}& =w_t-{\mathbf{H}}^{-1}\mathbf{g}\\ & =w_t-{\left({XBX}^{\top }\right)}^{-1}\mathbf{X}\left(p\left({\mathbf{X}}^{\top }\mathbf{w}\right)-\mathbf{Z}\right)\\ & ={\left({XBX}^{\top }\right)}^{-1}{XBr}_t\end{array}\text{\hspace{1em}(3.10)}$$

有${\mathbf{r}}_t={\mathbf{X}}^{\top }{\mathbf{w}}_t-{\mathbf{B}}^{-1}\left(p\left({\mathbf{X}}^{\top }{w}_t\right)-z\right)$ . This is the solution to the weighted least squares problem$\arg {\min }_{\mathbf{w}}.{\sum }_i{b}_i{\left(r_i-{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)}^2$.

Exercise: Prove that equation (3.7) and equation (3.3) in the scalar factor $Within\; 1/n$ are equivalent.

---

proof

首先，注意$\log \left(\sigma \left({\mathbf{w}}^{\top }\mathbf{x}\right)\right)=-\log \left(1+\exp \left(-{\mathbf{w}}^{\top }\mathbf{x}\right)\right)$ 和$\log \left(1-\sigma \left({\mathbf{w}}^{\top }\mathbf{x}\right)\right)=-\log \left(1+\exp \left({\mathbf{w}}^{\top }\mathbf{x}\right)\right)$。因此，公式（3.7）等价于

$$L(\mathbf{w})=\sum _{i=1}^n{z}_i\log \left(1+\exp \left(-{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)+\left(1-z_i\right)\log \left(1+\exp \left({\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)\text{\hspace{1em}(3.11)}$$

由于$z_i$不是0就是1，我们可以将和改写为

$$L(\mathbf{w})=\sum _{z_i=1}\log \left(1+\exp \left(-{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)+\sum _{z_i=0}\log \left(1+\exp \left({\mathbf{w}}^{\top }{x}_i\right)\right).\text{\hspace{1em}(3.12)}$$

Now, change the label $z_i$With the label yi y_{i} in (3.3)$y_i$In comparison, we see that $z_i=1\iff y_i=1$，$z_i=0\iff y_i=-1$，所以

$$\begin{array}{rl}L(\mathbf{w})& =\sum _{y_i=1}\log \left(1+\exp \left(-y_i{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)+\sum _{y_i=-1}\log \left(1+\exp \left(-y_i{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right)\\ & =\sum _{i=1}^n\log \left(1+\exp \left(-y_i{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)\right),\end{array}\text{\hspace{1em}(3.13)}$$

与方程(3.3)相吻合，其系数为$1/n$。

3.2 线性可分性和逻辑回归

需要注意的是，逻辑回归在线性可分离的训练集上可以过度拟合。当两个类1和$-1$是线性可分离的，我们可以找到一个超平面$\left({\mathbf{w}}_s,b_s\right)$，使得以下不等式成立。

$$y_i\left({\mathbf{w}}_s^{\top }{\mathbf{x}}_i+b_s\right)>0\forall i.\text{\hspace{1em}(3.14)}$$
考虑以下定理。

定理3.4

对于线性可分离的、非空的训练集，损失函数

$$F(\mathbf{w},b)=\sum _{i=1}^n\log \left(1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)\right)\text{\hspace{1em}(3.15)}$$

At ${\mathbb{R}}^{}$There is no global minimum in${}^p$${}^{+}$${}^{1\; .}$

---

Proof
First let us describe FFThe global minimum of$F.$It is$\left({\mathbf{w}}^{\ast },b^{\ast }\right)\in R^{}$A point in${}^p$${}^{+}$${}^{1\; ,\; that\; is}$

$$F(\mathbf{w},b)\ge F\left(w^{\ast },b^{\ast }\right)\forall (\mathbf{w},b)\in R^{p+1}\text{\hspace{1em}( 3.16 )}$$
exists$$\text{\hspace{1em}.}$$$$\text{\hspace{1em}\_}$$$$\text{\hspace{1em}\_}$$If the training set is non-empty, then the loss function is strictly positive definite. Therefore,$The\; minimum\; value\; of\; F$ is some positive number$eh$，即

$$F\left({\mathbf{w}}^{\ast },b^{\ast }\right)=e>0.\text{\hspace{1em}(3.17)}$$

Now we will prove the absence of a global minimum by means of a contradiction. Suppose there is a point $\left(w_s,b_s\right)$ and a real number$e>0$ , making the following conditions true.

$$y_i\left({\mathbf{w}}_s^{\top }{\mathbf{x}}_i+b_s\right)>0\forall i\text{ and }F(\mathbf{w},b)\ge \epsilon \forall (\mathbf{w},b)\in {\mathbb{R}}^{p+1}.\text{\hspace{1em}(3.18)}$$

for each $i$，定义以下标量值
$$X_i=y_i\left({\mathbf{w}}_s^{\top }{\mathbf{x}}_i+b_s\right)\text{\hspace{1em}(3.19)}$$

Consider the following equation

$$f_i(h)=\log \left(1+\exp \left(-h{\xi }_i\right)\right)\text{\hspace{1em}(3.20)}$$

It is easy to see that $x{i}_i$for any $i$ are strictly positive numbers, so$f_i(h)$随着$h$接近$\infty$而接近0。如果我们考虑$i$的总和，即

$$\underset{h\to \infty }{\lim }\sum _{i=1}^n{f}_i(h)=F\left(h{\mathbf{w}}_s,h{b}_s\right)=0.\text{\hspace{1em}(3.21)}$$

换句话说，对于任何$\epsilon >0$，我们可以找到一个实数$\eta >0$，这样，对于任何$h\ge \eta$，下面的不等式都成立。
$$F\left(h{\mathbf{w}}_s,h{b}_s\right)<\epsilon \text{\hspace{1em}(3.22)}$$

这直接与假设（3.18）相矛盾，因为我们可以选择$\mathrm{w}=h{\mathrm{w}}_s$和$b=h{b}_s$，且有$h\ge \eta$。

证明表明，一旦找到一个分离超平面，损失函数的值总是可以通过增加超平面参数的大小而进一步降低。请注意，这对任何分离类的超平面都是真实的。在实践中，一个优化算法可以选择一个具有 "坏 "位置和方向的超平面，并增加参数的大小，直到达到最大迭代次数。

为了防止这种情况，通常通过引入正则器来惩罚$(\mathbf{w},b)$的大小，例如通过固定一个实数常数$\lambda >0$并将原始成本函数调整为

$$\tilde{F}(\mathbf{w},b)=F(\mathbf{w},b)+\lambda \left(\Vert \mathbf{w}{\Vert }^2+b^2\right)\text{\hspace{1em}(3.23)}$$

3.3 逻辑回归的额外内容

任务1. 考虑为数据样本 y ∈ { − 1 , 1 } y\in\{-1,1\} y∈{ −1,1}分配一个标签的二元分类问题。通过逻辑回归的方式对数据样本 y ∈ { − 1 , 1 } y \in\{-1,1\} y∈{ −1,1 } Perform binary classification. Given a training set{ ( x 1 , y 1 ) , … , ( x N , y N ) } \left\{\left(\mathbf{x}_{1}, y_{1}\right), \ ldots,\left(\mathbf{x}_{N}, y_{N}\right)\right\}{ (x1​,y1​),…,(xN​,yN​) } tag data. To recap, the loss function is given by

$$L(\mathbf{w},b)=\sum _{i=1}^N\log \left(1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)\right)$$

3.3.1 Gradient ${\nabla }_{\mathbf{w},b}L$

Solution: According to the chain rule, we have

$$\begin{array}{rl}{\nabla }_{b}L& =\sum _{i=1}^N\frac{{\nabla }_b\left(1+\exp \left(-y_i\left(w^{\top }{x}_i+b\right)\right)\right)}{1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)}\\ & =-\sum _{i=1}^N{y}_i\frac{\exp \left(-y_i\left(w^{\top }{x}_i+b\right)\right)}{1+\exp \left(-y_i\left(w^{\top }{x}_i+b\right)\right)}\\ & =-\sum _{i=1}^N\frac{y_i}{1+\exp \left(y_i\left(w^{\top }{x}_i+b\right)\right)}\end{array}$$

Accordingly, we get

$$\begin{array}{rl}{\nabla }_{\mathbf{w}}L& =\sum _{i=1}^N\frac{1}{1+\exp \left(-y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i+b\right)\right)}{\nabla }_w\left(1+\exp \left(-y_i\left(w^{\top }{x}_i+b\right)\right)\right)\\ & =-\sum _{i=1}^N{y}_i\frac{\exp \left(-y_i\left(w^{\top }{x}_i+b\right)\right)}{1+\exp \left(-y_i\left(w^{\top }{x}_i+b\right)\right)}{\mathbf{x}}_i\\ & =-\sum _{i=1}^N\frac{y_i}{1+\exp \left(y_i\left(w^{\top }{x}_i+b\right)\right)}{x}_i\end{array}$$

3.3.2 Global Minimum of Loss Function

Assume that the two classes of the training set are linearly separable, that is, there is a weight vector ${\mathbf{w}}_s\in {\mathbb{R}}^p$ , and there is a bias$b_s$，从而
$$y_i\left(w_s^{\top }{x}_i+b_s\right)>0\forall i$$
holds. Prove that under this assumption, the loss function is$\left({\mathbf{w}}^{\ast },b^{\ast }\right)\in {\mathbb{R}}^{}$There is no global minimum in${}^p$${}^{+}$${}^{1\; .}$

Solution:
$The\; global\; minimum\; of\; L$ refers to$\left(w^{\ast },b^{\ast }\right)\in R^{}$A pair in${}^p$${}^{+}$${}^1$
$$L(\mathbf{w},b)\ge L\left(w^{\ast },b^{\ast }\right)\forall (\mathbf{w},b)\in R^{p+1}$$
holds. Furthermore, for a non-empty training set,$L$ is strictly positive, so we can conclude that
$$L\left({\mathrm{w}}^{\ast },b^{\ast }\right)=e>0$$
Assume that such a point exists. Let us define
$$z_i=y_i\left(w_s^{\top }{\mathbf{x}}_i+b_s\right)$$
Please note that$z_i$for every $i$ are all strictly positive numbers. Consider the function
$$f(h)=\sum _{i=1}^N\log \left(1+\exp \left(-h{z}_i\right)\right)$$
Since every sum follows$h$ is close to$\infty$ and close to 0,f ( h ) f(h)The same is true for$f$$($$h$$)$
$$\underset{h\to \infty }{\lim }f(h)=0Observe$$
the equal case
$$f\left(h\right)=L\left(h{\mathbf{w}}_{\mathbf{s}},h{b}_s\right)$$
This means that for any$e>0$ , we can find inR \mathbb{R}Find ahh$in\; R$$h$，并设定$(\mathbf{w},b)=\left(h{\mathrm{w}}_{\mathrm{s}},h{b}_s\right)$ ,let
$$L(\mathbf{w},b)<\epsilon$$
is established, which is consistent with$\left({\mathbf{w}}^{\ast },b^{\ast }\right)$contradicts the assumption,$L\left(w^{\ast },b^{\ast }\right)=\epsilon$ is a global minimum.

Note that $\left(w_s,b_s\right)$ is not necessarily optimal in any sense. Depending on the algorithm, this may result in a "non-ideal" hyperplane descriptor with increasing constants.

3.3.3 Overfitting

To avoid the situation in 3.3.2, one can penalize (w, b) (w, b) by adding a square norm controllerThe norm of$($$w$$,$$b$$)\; .$Consider the modified loss function
$$\tilde{L}(\mathbf{w},b)=L(\mathbf{w},b)+l\left(\Vert \mathbf{w}{\Vert }^2+b^2\right)$$
where$l>0$ is a real-valued constant. Calculate the gradient${\nabla }_{\mathbf{w},b}\tilde{L}$ 。

Solution:
Due to the linear nature of the derivative, we have
$${\nabla }_b\tilde{L}={\nabla }_{b}L+2\lambda b$$
和
$${\nabla }_{\mathrm{w}}\tilde{L}={\nabla }_{w}L+2{min}_w$$

3.4 Logistic regression example

The following exercise is based on the extracted feature representation, but we will implement logistic regression manually, that is, by minimizing $F\left(w\right)$ instead of a pre-built classifier. To do this, make sure the variablestrain,test,train_data_featuresandtest_data_features(from the extracted features) are loaded into your IPython shell.

a) Write a PYTHON function logistic_gradient, expecting to train the training set matrix X_trainand the ground truth label vector $y_{\text{train}}$and the current weight vector $w$ takes as its input and returns the gradient$g$ . For mathematical definitions, please refer to the handout.

b) Write a PYTHON function find_wthat expects a training set matrix X_train, a ground truth label vector y_train, a step size α and a maximum number of iterations max_itto determine the optimal logistic regression weight vector by performing gradient descent w_star, i.e. calling logistic_gradient in each iteration . Make sure to set the affine offset $w_0=bIncorporate$ into your model.

c) The data set at hand is quite large. Applying standard gradient descent may cause Python to throw a MemoryError exception. To avoid this, we will use a variation of stochastic gradient descent that has proven successful in training deep neural networks. In minibatch learning, each iteration of the algorithm is epochreplaced by a so-called At each epoch, the training set is randomly divided into equal-sized subsets, called mini-batches. For each minibatch, the gradient is calculated and applied only to the samples in the minibatch. epochOne ends when each minibatch's gradient step is executed . Modify find_w to enable small-batch learning. You need to use $n_{-}$epochs replaces max_it and adds parameter n_minibatch in the function definition. Note: Pay attention to the normalization of gradient and loss functions.

d) Write a function classify_logwith a weight vector wand a test set matrix , classify the samples X_testthrough logistic regression , and return a label vector y test. y_{\text {test. }}X_test$y_{\text{test. }}$Test implementation of sum on and over train_data_features10 epochs, s size 100, step size .test_data_featuresfind_wclassify_logminibatchealpha=1

e) Logistic regression is prone to overfitting. To prevent this, a regularizer can be used. Adjust the implementation so that it does not minimize $F\left(w\right)$ , but the minimization term
$$F(\mathbf{w})+\lambda \Vert \mathbf{w}{\Vert }^{2where\lambda }$$
\$\lambda$ is a non-negative regularization parameter. Use$l=10^{-3}$ Test implementation.

3.4.1 Implementation using python

The relevant python code is as follows

import numpy as np
import pandas as pd
from bs4 import BeautifulSoup
import re
import nltk
import matplotlib.pyplot as plt

nltk.download('stopwords')  # Download text data sets, including stop words
from nltk.corpus import stopwords  # Import the stop word list
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression as LR
from sklearn.metrics import roc_auc_score as AUC


# function for preprocessing the data
def review_prepro(data, remove_stopwords=False):
    stops = stopwords.words('english')
    # remove HTML tags
    review_text = BeautifulSoup(data, 'lxml').get_text()
    # remove non-letters and numbers
    letters_only = re.sub('[^a-zA-Z]',
                          ' ',
                          review_text)
    # make all characters lower case and split the documents into single words
    words = letters_only.lower().split()

    if remove_stopwords:
        # remove stop words
        meaningful_words = [w for w in words if not w in stops]
        # return concatenated single string
        return ' '.join(meaningful_words)
    else:
        # or don't and concatenate to single string
        return ' '.join(words)


def classify_log(w, X_test):
    w0 = w[0]
    y_test = sigmoid(w0 + np.dot(X_test.T, w[1:]))
    return y_test


def train_data_prep(train, vectorizer):
    """
    preprocess the training data using the bag of words in Sklearn
    :param train:
    :return: processed training data
    """
    # load train data
    num_reviews = train['review'].size
    clean_train_reviews = []
    for i in range(num_reviews):
        if (i + 1) % 1000 == 0:
            print('\r Review {} of {} - Training'.format(i + 1, num_reviews), end="")
        clean_train_reviews.append(review_prepro(train['review'][i], remove_stopwords=True))

    # fit the vectorizer to the data
    train_data_features = vectorizer.fit_transform(clean_train_reviews)
    # convert to numpy array
    train_data_features = train_data_features.toarray()

    return train_data_features


def test_data_prep(test, vectorizer):
    """
    preprocess the testing data from the raw input
    :param test:
    :return: processed testing data
    """
    num_test_reviews = test['review'].size
    clean_test_reviews = []
    for i in range(num_test_reviews):
        if (i + 1) % 1000 == 0:
            print('\r Review {} of {}'.format(i + 1, num_test_reviews), end='')
        clean_test_reviews.append(review_prepro(test['review'][i], remove_stopwords=True))

    test_data_features = (vectorizer.transform(clean_test_reviews)).toarray()
    return test_data_features


def sigmoid(x):
    # https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/
    z = np.exp(-np.abs(x))
    return np.where(x >= 0.0, 1.0 / (1.0 + z), z / (1.0 + z))


def logistic_gradient(x_train, y_train, w, reg=0.0):
    g = -np.dot(x_train * y_train, sigmoid(-np.dot(w, x_train) * y_train)) / x_train.shape[1] + reg * np.sum(w ** 2)
    return g


def find_w(x_train, y_train, alpha, n_epochs, n_minibatch, reg=0.0):
    """
    Using this function to find the best w.
    :param x_train: the training data, which has the shapes [samples, features]
    :param y_train: the training label, which has the shapes [outputs,]
    :param alpha: step constant
    :param n_epochs: training epochs
    :param n_minibatch: we divided the training data into some small training set to accelerate the training speed.
    :param reg: the factor of the regularizer
    :return: the best weight w_star
    """
    x_train = x_train.T
    x_pre = np.ones((x_train.shape[0] + 1, x_train.shape[1]))
    x_pre[1:, :] = x_train
    w = np.ones((x_pre.shape[0],))
    print('x_pre shape ={}, w shape = {}'.format(x_pre.shape, w.shape))
    loss_ = []
    for k in range(n_epochs):
        loss = np.sum(-np.log(sigmoid(y_train * np.dot(w, x_pre)))) / x_pre.shape[1] + reg * np.sum(w ** 2)
        loss_.append(loss)
        rp = np.random.permutation(x_pre.shape[1])
        print('In {} iteration, the loss = {}'.format(k, loss))
        for it in range(x_pre.shape[1] // n_minibatch):
            delta_w = logistic_gradient(x_pre[:, rp[it * n_minibatch: (it + 1) * n_minibatch]],
                                        y_train[rp[it * n_minibatch:(it + 1) * n_minibatch]], w)
            w = w - alpha * delta_w
    return w,loss_


if __name__ == "__main__":
    # load the data
    train = pd.read_csv('labeledTrainData.tsv', header=0, delimiter='\t', quoting=3)
    test = pd.read_csv('labeledTestData.tsv', header=0, delimiter="\t", quoting=3)

    # download the stopwords
    stops = set(stopwords.words('english'))
    vectorizer = CountVectorizer(analyzer='word',
                                 tokenizer=None,
                                 preprocessor=None,
                                 stop_words=stops,
                                 max_features=5000)

    # process the data
    train_data_features = train_data_prep(train, vectorizer)
    test_data_features = test_data_prep(test, vectorizer)
    y_train = (np.array(train['sentiment'].values) - 0.5) * 2

    # (1) Testing implementation without regularizers
    W, W_loss = find_w(train_data_features, y_train, 0.8, 20, 100)
    y_pred = classify_log(W, test_data_features.T)
    y_test = test['sentiment'].values
    auc = AUC(y_test, y_pred)
    print('AUC score after 20 epochs:', auc)

    # (2) Testing implementation with regularizers to overdue mitigate the overfitting
    w, w_loss = find_w(train_data_features, y_train, 1, 20, 100, 1e-3)
    y_pred = classify_log(w, test_data_features.T)
    auc = AUC(y_test, y_pred)
    print('AUC score after 20 epochs:', auc)

    # plot
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.set_xlabel('Iteration k')
    ax.set_ylabel('loss')
    ax.plot(W_loss, marker='.', label='without regularizers')
    ax.plot(w_loss, marker='.', label='with regularizers')
    ax.legend()
    plt.show()

The output is,

x_pre shape =(5001, 20000), w shape = (5001,)
In 0 iteration, the loss = 48.15060140614601
In 1 iteration, the loss = 1.9271375244901945
In 2 iteration, the loss = 1.1439895026827396
In 3 iteration, the loss = 0.8040511998178164
In 4 iteration, the loss = 0.637276530141581
In 5 iteration, the loss = 0.5275615696559821
In 6 iteration, the loss = 0.4531238182560532
In 7 iteration, the loss = 0.39835683526227206
In 8 iteration, the loss = 0.3585725270966355
In 9 iteration, the loss = 0.3208458216978429
In 10 iteration, the loss = 0.2989468342660105
In 11 iteration, the loss = 0.27127014988315357
In 12 iteration, the loss = 0.26380395477375235
In 13 iteration, the loss = 0.23870379629686295
In 14 iteration, the loss = 0.22874535039441457
In 15 iteration, the loss = 0.22097269464251518
In 16 iteration, the loss = 0.21411424494393824
In 17 iteration, the loss = 0.20234804193010983
In 18 iteration, the loss = 0.19146457065053046
In 19 iteration, the loss = 0.18977771952092898
AUC score after 20 epochs: 0.9201902430149848

x_pre shape =(5001, 20000), w shape = (5001,)
In 0 iteration, the loss = 53.15160140614601
In 1 iteration, the loss = 4.850572401123086
In 2 iteration, the loss = 3.7359586329298007
In 3 iteration, the loss = 3.2139682944306305
In 4 iteration, the loss = 2.907177612020046
In 5 iteration, the loss = 2.681058512521938
In 6 iteration, the loss = 2.5166307885635284
In 7 iteration, the loss = 2.392833861337298
In 8 iteration, the loss = 2.294087922661967
In 9 iteration, the loss = 2.2217974363161987
In 10 iteration, the loss = 2.161302804041983
In 11 iteration, the loss = 2.113409966477404
In 12 iteration, the loss = 2.0736801266948683
In 13 iteration, the loss = 2.061541837823529
In 14 iteration, the loss = 2.016740023535874
In 15 iteration, the loss = 1.9973296285347977
In 16 iteration, the loss = 1.9823663669782907
In 17 iteration, the loss = 1.967645885882539
In 18 iteration, the loss = 1.9572636195387196
In 19 iteration, the loss = 1.9538835403945283
AUC score after 20 epochs: 0.921975893241498