Andrew Ng's "Machine Learning" - Logistics regression code implementation

The data set and source files can be obtained in the Github project
link: https://github.com/Raymond-Yang-2001/AndrewNg-Machine-Learing-Homework

1. Sigmoid and binary classification

Unlike linear regression, logistic regression, although called regression, is often used to implement classification functions. The only difference from linear regression is that a Sigmoid function is added after the output of linear regression so that its output can represent the probability of classification without predicting the value. As for why the Sigmoid function has such a function, we will explain it next.

Sigmoid function

The mathematical expression of the Sigmoid function is as follows:
$$\sigma \left(x\right)=\frac{1}{1+e^{-x}}$$
Analysis can show that the Sigmoid function is an increasing function. The larger the x value, $The\; closer\; \sigma \left(x\right)$ is to 1, the smaller the x value is,$The\; closer\; \sigma \left(x\right)$ is to 0. When x=0,$\sigma \left(x\right)=0.5$ . Its function curve is as follows:

Why can the Sigmoid function represent binary classification probability?

Here we first review the knowledge of Bernoulli distribution .

Bernoulli distribution refers to the random variable $X$ has, the parameter is$p(0<p<1)$ , if it is respectively with probability$p$ and$1-p$ takes values ​​1 and 0. $E\left(X\right)=p$,$D(X)=p(1-p)$ . The number of successful Bernoulli trials obeys the Bernoulli distribution, parameter$p$ is the probability of success of the experiment. The Bernoulli distribution is a discrete probability distribution, which is$N=1.$ Special case of binomial distribution

For the Bernoulli distribution, we have $p=m^x(1-m{)}^{(1-x)}$，对等式做电影，有
$$p=e^{x\ln m+(1-x)\ln (1-\mu )}=e^{x\ln \frac{}{1-m}+\ln (1-\mu )}$$
Next, we usethe exponential family distributionto represent the Bernoulli distribution.

The exponential family of distributions, also known as the exponential distribution family, is the most important parameter distribution family in statistics.

The general parameterization of the exponential family distribution is expressed as:
$$p(y;the)=b(y)e^{{the}^{\top }T\left(y\right)-\alpha \left(\eta \right)}$$
among them，

• $y$ is a natural parameter
• $T\left(y\right)$是yySufficient statistics of$y$
• $\alpha \left(\eta \right)$ is the logarithmic part function, used to ensure that$\sum p(y;the)=1$

From this formula, we can get, $the=\ln \frac{}{1-m}$，即
$$m=\frac{1}{1+e^{-h}}$$

It can be seen that the Sigmoid function can express the probability of Bernoulli distribution.

2. Logistics regression

Like linear regression, logistic regression can also be solved using gradient-based optimization methods. The model formula of logistic regression is as follows:
$$h(\mathbf{x};i)=\sigma \left(\theta x^{\top }\right)$$
where,$x$ is a dimension of$(n,d+1)$ sample (the first dimension is added to be all 1 to facilitate the calculation of the offset term),$\theta$ is$(1,d+1)$ parameters. Get$(1,n)$ output.

Cross entropy loss function

In linear regression, we can use the mean square error MSE to measure the difference between the predicted value and the true value. This loss function based on the numerical difference is easy to understand. However, in classification tasks, MSE is obviously no longer suitable for measuring classification differences. For this reason, we introduce a new loss function-cross entropy loss.

In order to understand cross-entropy loss, we first start with an important concept in information theory- KL divergence .

In classification tasks, whether it is multi-classification or two-classification, our task can be seen as outputting a predicted distribution . For binary classification, this is a Bernoulli distribution, and for multiclass classification, this is a multinomial distribution. The better the classification effect, the closer the output distribution should be to the target distribution. So how do you measure the similarity of two distributions? This is what KL divergence does.

Consider a $Classification\; problem\; of\; K$ class, let our target distribution be$q(k|x)$ , the output distribution is$p(k|x)$ . These two distributions respectively specify that the sample is the$The\; probability\; of\; k$ classes. For the target distribution, it is obvious that this is a one-hot style distribution, that is, the probability of the true category is 1, and the other probabilities are 0. The KL divergence of the two distributions is written as:
$$KL(q||p)=\sum _{k=1}^{K}q(k|x)\log \frac{q(k|x)}{p(k|x)}$$
The closer the two distributions are, the smaller the KL divergence is. It can be observed that when $p=When\; q$ , the KL divergence is 0.

If we further expand the KL divergence, we can get:
$$KL(q||p)=\sum _{k=1}^{K}q(k|x\left)\log q\right(k|x)-q(k|x)\log$$
The first half of$$p$$$$($$$$k$$$$|$$$$x$$$$)$$ is about the distribution qqconstant for$q$$q$ is a fixed target distribution, and the KL divergence is only related to the second half, also called cross entropy.
$$CrossEntropy=-\sum _{k=1}^{K}q(k|x)\log$$
The closer the two distributions$$p$$$$($$$$k$$$$|$$$$x$$$$)\; are,\; the\; smaller\; the\; cross\; entropy\; is,\; and\; conversely,\; the\; larger\; the\; cross\; entropy\; is.$$

In particular, for binary classification tasks, there is binary cross entropy BCE (Binary Cross Entropy):
$$BCE=-(y\log \hat{y}+(1-y)\log (1-\hat{y}\left)\right)$$
This is actually$K=2$ special circumstances.

gradient

The gradient representation of logistic regression is the same as linear regression, both are
$$i_j=i_j-a\frac{1}{m}\sum _{i=1}^m(h(x^{(i)};i)-y^{(i)})x_j^{(i)}$$
Note that in logistic regression, $h(\mathbf{x};i)=\sigma \left(\theta x^{\top }\right)$, in linear regression, there is no Sigmoid operation.

The specific derivation is as follows:
$$J(\mathbf{\theta })=-\left[y\log p+(1-y)\log (1-p)\right]$$
index,$p=\sigma (\mathbf{\theta x})$。
由线性法则：
$$\frac{\partial J}{\partial {\theta }_j}=\frac{\partial J}{\partial p}\frac{\partial p}{\partial (\mathbf{\theta x})}\frac{\partial (\mathbf{\theta x})}{\partial {\theta }_j}$$
in:

• $\frac{\partial J}{\partial p}=-\frac{y}{p}+\frac{1-y}{1-p}$
• $\frac{\partial p}{\partial (\mathbf{\theta x})}=\sigma \left(\theta x\right)(1-\sigma (\theta x\left)\right)$
• $\frac{\partial (\mathbf{\theta x})}{\partial {\theta }_j}=x_j$

For one equation, let
$\begin{array}{rl}J(\mathbf{\theta })& =\left[-\frac{y}{p}+\frac{1-y}{1-p}\right]\sigma \left(\theta x\right)(1-\sigma (\theta x\left)\right)x_j\\ & =\left[-\frac{y}{\sigma \left(\theta x\right)}+\frac{1-y}{1-\sigma \left(\theta x\right)}\right]\sigma \left(\theta x\right)(1-\sigma (\theta x\left)\right)x_j\\ & =\left[-y(1-\sigma (\theta x))+(1-y)\sigma \left(\theta x\right)\right]x_j\\ & =\left(\sigma \right(\theta x)-y)x_j\end{array}$

Overfitting and underfitting

In the field of machine learning, overfitting has always been an important problem. Overfitting refers to the situation where the learned model has small deviation and excessive variance .

For example, there is a large classification data set, the labels of this data set conform to a certain distribution, and the goal of the model is to learn the mapping from samples to this distribution. Suppose we randomly divide this data set into 10 subsets and train the corresponding ten models respectively. These ten models should be able to classify well on their own data sets, and the gap between the models should not be Big - because all the data comes from one dataset. This means that the bias and variance of the model are relatively small .

Consider a situation, assuming that the "dog" samples included in one of the data sets are all white dogs, this model believes that all white animals are dogs. Although this can classify well on the "White Dog" data set, compared with other models, the "difference" of this model appears to be too large. This means that when the deviation is small, the variance becomes large . Also called overfitting.

On the contrary, if the deviation is large and the variance is small , that is, the differences between the models are very small, but they cannot be accurately classified, this situation is called underfitting .

For underfitting, we can solve it through data enhancement or violently increasing the number of iterations. For overfitting, we will introduce a method called regularization.

Regularization

The most commonly used regularization method is to append a parameter penalty term after the loss function to control the parameters from developing in the direction of overfitting. The general expression form is:
$$Regularizedloss=Loss+\lambda \left(\theta \right)$$
In this code, we implement$L^2$正则化。
正则化损失函数如下所示：
$$J\left(i\right)=\frac{1}{m}\sum _{i=1}^m[-y^{(i)}\log \left(h_{}\left(x^{(i)}\right)\right)-\left(1-y^{(i)}\right)\log \left(1-h_{}\left(x^{(i)}\right)\right)]+\frac{}{2m\_}\sum _{j=1}^n{i}_j^2$$
We find the gradient and get the regularized gradient:
$$g\left(i\right)=\frac{1}{m}\sum _{i=1}^m(h(x^{(i)};i)-y^{(i)})x_j^{(i)}+\frac{}{m}{i}_j$$
In the case of the infinitesimal equation, we have the following formula:
$$i_j=i_j-a\frac{1}{m}\sum _{i=1}^m\left(h\right(x^{\left(i\right)};i)-y^{\left(i\right)})x_j^{\left(i\right)}-\frac{}{m}\sum _{j=1}^n{i}_j=\left(1-\frac{}{m}\right)i_j-a\frac{1}{m}\sum _{i=1}^m\left(h\right(x^{\left(i\right)};i)-y^{\left(i\right)})x_j^{\left(i\right)}$$
It can be seen that regularization actually reduces the parameters to a certain extent. The degree of reduction is related to $related\; to\; \lambda /m$ .

This figure illustrates$L^2$ Influence of regularization parameters. The dashed line represents the loss contour of the regularized term, and the solid line represents the loss contour of the unregularized loss function. Both are in$\tilde{w}$ Achieve balance. at$w_1$direction, when the parameters change, the loss function does not change too much, but when $w_2$This change appears to be more dramatic. That is to say $w_2$Compared $w_1$It can significantly reduce the loss function value.

$L^2$ Regularization makes the parameters in the direction of significantly reducing the value of the loss function better preserved, and does not help the direction of the loss function to change significantly, because this will not significantly affect the gradient.

3. Python code implementation

Here is the code for the Logistic regression class, which also implements data normalization and regularization:

import numpy as np


def sigmoid(x):
    return 1 / (1 + np.exp(-x))


def bce_loss(pred, target):
    """
    计算误差
    :param pred: 预测
    :param target: ground truth
    :return: 损失序列
    """
    return -np.mean(target * np.log(pred) + (1-target) * np.log(1-pred))


class LogisticRegression:
    """
    Logistic回归类
    """

    def __init__(self, x, y, val_x, val_y, epoch=100, lr=0.1, normalize=True, regularize=None, scale=0, show=True):
        """
        初始化
        :param x: 样本, (sample_number, dimension)
        :param y: 标签, (sample_numer, 1)
        :param epoch: 训练迭代次数
        :param lr: 学习率
        """
        self.theta = None
        self.loss = []
        self.val_loss = []
        self.n = x.shape[0]
        self.d = x.shape[1]

        self.epoch = epoch
        self.lr = lr

        t = np.ones(shape=(self.n, 1))

        self.normalize = normalize

        if self.normalize:
            self.x_std = x.std(axis=0)
            self.x_mean = x.mean(axis=0)
            self.y_mean = y.mean(axis=0)
            self.y_std = y.std(axis=0)
            x = (x - self.x_mean) / self.x_std

        self.y = y
        self.x = np.concatenate((t, x), axis=1)

        # self.val_x = (val_x - val_x.mean(axis=0)) / val_x.std(axis=0)
        self.val_x = val_x
        self.val_y = val_y

        self.regularize = regularize
        self.scale = scale

        self.show = show

    def init_theta(self):
        """
        初始化参数
        :return: theta (1, d+1)
        """
        self.theta = np.zeros(shape=(1, self.d + 1))

    def gradient_decent(self, pred):
        """
        实现梯度下降求解
        """
        # error (n,1)
        error = pred - self.y
        # term (d+1, 1)
        term = np.matmul(self.x.T, error)
        # term (1,d+1)
        term = term.T

        if self.regularize == "L2":
            re = self.scale / self.n * self.theta[0, 1:]
            re = np.expand_dims(np.array(re), axis=0)
            re = np.concatenate((np.array([[0]]), re), axis=1)
            # re [0,...] (1,d+1)
            self.theta = self.theta - self.lr * (term / self.n + re)
        # update parameters
        else:
            self.theta = self.theta - self.lr * (term / self.n)

    def validation(self, x, y):
        if self.normalize:
            x = (x - x.mean(axis=0)) / x.std(axis=0)
        outputs = self.get_prob(x)
        curr_loss = bce_loss(outputs, y)
        if self.regularize == "L2":
            curr_loss += self.scale / self.n * np.sum(self.theta[0, 1:] ** 2)
        self.val_loss.append(curr_loss)
        predicted = np.expand_dims(np.where(outputs[:, 0] > 0.5, 1, 0), axis=1)
        count = np.sum(predicted == y)
        if self.show:
            print("Accuracy on Val set: {:.2f}%\tLoss on Val set: {:.4f}".format(count / y.shape[0] * 100, curr_loss))

    def test(self, x, y):
        outputs = self.get_prob(x)
        predicted = np.expand_dims(np.where(outputs[:, 0] > 0.5, 1, 0), axis=1)
        count = np.sum(predicted == y)
        # print("Accuracy on Test set: {:.2f}%".format(count / y.shape[0] * 100))
        # curr_loss = bce_loss(outputs, y)
        # if self.regularize == "L2":
        # curr_loss += self.scale / self.n * np.sum(self.theta[0, 1:] ** 2)
        return count / y.shape[0]  # , curr_loss

    def train(self):
        """
        训练Logistic回归
        :return: 参数矩阵theta (1,d+1); 损失序列 loss
        """
        self.init_theta()

        for i in range(self.epoch):
            # pred (1,n); theta (1,d+1); self.x.T (d+1, n)
            z = np.matmul(self.theta, self.x.T).T
            # pred (n,1)
            pred = sigmoid(z)
            curr_loss = bce_loss(pred, self.y)
            if self.regularize == "L2":
                curr_loss += self.scale / self.n * np.sum(self.theta[0, 1:] ** 2)
            self.loss.append(curr_loss)
            self.gradient_decent(pred)
            if self.show:
                print("Epoch: {}/{}, Train Loss: {:.4f}".format(i + 1, self.epoch, curr_loss))
            self.validation(self.val_x, self.val_y)

        if self.normalize:
            y_mean = np.mean(z, axis=0)
            self.theta[0, 1:] = self.theta[0, 1:] / self.x_std.T
            self.theta[0, 0] = y_mean - np.dot(self.theta[0, 1:], self.x_mean.T)
        return self.theta, self.loss, self.val_loss

    def get_prob(self, x):
        """
        回归预测
        :param x: 输入样本 (n,d)
        :return: 预测结果 (n,1)
        """
        t = np.ones(shape=(x.shape[0], 1))
        x = np.concatenate((t, x), axis=1)
        pred = sigmoid(np.matmul(self.theta, x.T))
        return pred.T

    def get_inner_product(self, x):
        t = np.ones(shape=(x.shape[0], 1))
        x = np.concatenate((t, x), axis=1)
        return np.matmul(self.theta, x.T)

    def predict(self, x):
        prob = self.get_prob(x)
        return np.expand_dims(np.where(prob[:, 0] > 0.5, 1, 0), axis=1)

4. Single-dimensional and multi-dimensional logistic classification

Single-dimensional data classification

Data set visualization

divides training set and validation set.
Training set visualization:

Validation set visualization:

Call algorithm for classification.

from LogisticRegression import LogisticRegression

epochs = 5000
alpha = 0.01
logistic_reg = LogisticRegression(x=train_x,y=train_y_ex,val_x=val_x,val_y=val_y_ex,epoch=epochs,lr=alpha)
theta,train_loss,val_loss = logistic_reg.train()

Classification performance

Accuracy on Test Set: 80.00%
My F1 Score: 0.8571

sklearn library function verification

Sklearn Accuracy: 80.00%
Sklearn F1 Score: 0.8571

Visualizing decision boundaries

Visualization of the training process
shows that there is obvious overfitting, which can be suppressed by adjusting the learning rate and the number of iterations. Of course, regularization can also be performed.

Multidimensional data classification

The data set is visually

divided into training set and validation set
. Training set:

Validation set:

Do data enhancement (expand data dimensions)
$$\mathbf{X}=[x_1,x_2,x_1^2,x_1{x}_2,x_2^2,x_1^3,x_1^2{x}_2,\cdots ]$$
This is expanded to the sixth power

def feature_mapping(x, degree):
    feature = np.zeros([x.shape[0],1])
    for i in range(0, 1 + degree):
        for j in range(0, 1 + degree - i):
            if i==0 and j==0: continue
            feature=np.concatenate((feature, np.expand_dims(np.multiply(np.power(x[:, 0], i) , np.power(x[:, 1], j)), axis=1)),axis=1)
    return feature[:,1:]

train_x_map = feature_mapping(train_x,degree=6)
val_x_map = feature_mapping(val_x,degree=6)

The regularization parameter is 2

Accuracy on Test Set: 66.67%
My F1 Score: 0.5556
Sklearn Accuracy: 70.83%
Sklearn Val Loss: 0.3076
Sklearn F1 Score: 0.6667

The regularization parameter of the visualization of the decision boundary

is 0. It can be seen from the visualization of the decision boundary
that the decision boundary appears to be too "deformed", which is a prominent manifestation of over-fitting.