笔记：Logistic回归基本原理

app显示公式乱码，请使用浏览器或电脑查看。

笔记：Logistic回归基本原理

1、Bernoulli(亮点分布，0-1分布)

成功变量X取值1，否则取值0。成功概率为 $\theta$ ，我们称X服从参数为 $\theta$ 的Bernoulli分布，记作 $X \text{～} Ber(\theta)$

概率函数(pmf)为：

p (x) = θ^{x} (1 - θ)^{1 - x} = {\begin{cases} θ, & if x = 1 \\ 1 - θ, & if x = 0 \end{cases}

$p(x) = \theta^x (1 - \theta)^{1 - x} = \begin{cases} \theta, & \text{if $x = 1$} \\ 1 - \theta, & \text{if $x = 0$} \end{cases}$

均值： $\mu = \theta$
方差： $\sigma^2 = \theta \times (1 - \theta)$

2、Logistic回归模型

Logistic回归模型是一个线性模型

条件概率：

\begin{array}{ll} p (y | x) & = B e r (y | μ (x)) \\ μ (x) & = σ (w^{T} x) \end{array}

$\begin{array}{ll} p(y|{\bf x}) &= Ber(y|\mu ({\bf x})) \\[2ex] \mu({\bf x}) &= \sigma({\bf w^T x}) \end{array}$

sigmoid函数（S形函数）：

σ (a) = \frac{1}{1 + e x p (- a)} = \frac{e x p (a)}{e x p (a) + 1}

$\sigma(a) = \frac{1}{1 + exp(-a)} = \frac{exp(a)}{exp(a) + 1}$

亦被称为logistic函数或logit函数，将实数a变换到[0, 1]区间

3、神经科学中的Logistic回归

神经元对其输入进行加权和： $\bf f(x) = w^T x$
在Logistic回归，定义Log Odds Ratio:

\begin{array}{ll} L O R (x) & = \log \frac{p (y = 1 | x, w)}{p (y = 0 | x, w)} \\ = \log [\frac{1}{1 + e x p (- w^{T} x)} \times \frac{1 + e x p (- w^{T} x)}{e x p (- w^{T} x)}] \\ = \log [e x p (w^{T} x)] \\ = w^{T} x \end{array}

$\bf \begin{array}{ll} LOR(x) &= \log{\frac{p(y = 1 | x, w)}{p(y = 0 | x, w)}} \\[2ex] &= \log{ \left[ \frac{1}{1 + exp(-w^T x)} \times \frac{1 + exp(-w^T x)}{exp(-w^T x)} \right] } \\[2ex] &= \log{ \left[ exp(w^T x) \right] } \\[2ex] &= w^T x \end{array}$

$iff \quad \bf LOR(x) = w^T x > 0$ ，神经元发放脉冲，即

p (y = 1 | x, w) > p (y = 0 | x, w)

$p(y = 1 | {\bf x, w}) > p(y = 0 | {\bf x, w})$

4、线性决策函数

在Logistic回归中

\begin{array}{ll} L O R (x) & = w^{T} x > 0, & \hat{y} = 1 \\ L O R (x) & = w^{T} x < 0, & \hat{y} = 0 \\ L O R (x) & = w^{T} x = 0 : & 决策面 \end{array}

$\bf \begin{array}{ll} LOR(x) &= w^T x > 0, & \hat{y} = 1 \\[2ex] LOR(x) &= w^T x < 0, & \hat{y} = 0 \\[2ex] LOR(x) &= w^T x = 0: & \text{决策面} \end{array}$

因为Logistic回归是一个线性分类器

5、极大似然估计（损失函数）

令 $\mu_i = \mu({\bf x}_i)$ ，则负 $\log$ 似然为：

\begin{array}{ll} J (w) = N L L (w) & = - \sum_{i = 1}^{N} \log [(μ_{i})^{y_{i}} \times (1 - μ_{i})^{1 - y_{i}}] \\ = \sum_{i = 1}^{N} - [y_{i} \log (μ_{i}) + (1 - y_{i}) \log (1 - μ_{i})] \end{array}

$\begin{array}{ll} J({\bf w}) = NLL({\bf w}) &= - \sum_{i = 1}^{N} \log{ \left[ (\mu_i)^{y_i} \times (1 - \mu_i)^{1 - y_i} \right] } \\[2ex] &= \sum_{i = 1}^{N} - \left[ y_i \log(\mu_i) + (1 - y_i) \log(1 - \mu_i) \right] \end{array}$

极大似然估计等价于最小Logistic损失

6、正则

目标函数

$J (w) = \sum_{i = 1}^{N} - [y_{i} \log (μ_{i}) + (1 - y_{i}) \log (1 - μ_{i})]$ $J({\bf w}) = \sum_{i = 1}^{N} - \left[ y_i \log(\mu_i) + (1 - y_i) \log(1 - \mu_i) \right]$
L2

$J (w) = \sum_{i = 1}^{N} - [y_{i} \log (μ_{i}) + (1 - y_{i}) \log (1 - μ_{i})] + λ | | w | |_{2}^{2}$ $J({\bf w}) = \sum_{i = 1}^{N} - \left[ y_i \log(\mu_i) + (1 - y_i) \log(1 - \mu_i) \right] + \lambda ||{\bf w}||_2^2$
L1

$J (w) = \sum_{i = 1}^{N} - [y_{i} \log (μ_{i}) + (1 - y_{i}) \log (1 - μ_{i})] + λ | w |$ $J({\bf w}) = \sum_{i = 1}^{N} - \left[ y_i \log(\mu_i) + (1 - y_i) \log(1 - \mu_i) \right] + \lambda |{\bf w}|$

7、优化方法

目标函数

$J (w) = \sum_{i = 1}^{N} - [y_{i} \log (μ_{i}) + (1 - y_{i}) \log (1 - μ_{i})]$ $J({\bf w}) = \sum_{i = 1}^{N} - \left[ y_i \log(\mu_i) + (1 - y_i) \log(1 - \mu_i) \right]$
梯度

$\begin{array}{ll} g (w) = \frac{\partial J (w)}{\partial w} & = \frac{\partial}{\partial w} [\sum_{i = 1}^{N} - [y_{i} \log (μ_{i}) + (1 - y_{i}) \log (1 - μ_{i})]] \\ = \sum_{i = 1}^{N} [- y_{i} \times \frac{1}{μ (x_{i})} + (1 - y_{i}) \times \frac{1}{1 - μ (x_{i})}] \frac{\partial}{\partial w} μ (x_{i}) \\ = \sum_{i = 1}^{N} [- y_{i} \times \frac{1}{μ (x_{i})} + (1 - y_{i}) \times \frac{1}{1 - μ (x_{i})}] μ (x_{i}) (1 - μ (x_{i})) x_{i} \\ = \sum_{i = 1}^{N} [- y_{i} \times [1 - μ (x_{i})] + (1 - y_{i}) μ (x_{i})] x_{i} \\ = \sum_{i = 1}^{N} [- y_{i} + μ (x_{i})] x_{i} \\ = \sum_{i = 1}^{N} [μ (x_{i}) - y_{i}] x_{i} \end{array}$ $\begin{array}{ll} g({\bf w}) = \frac{\partial J({\bf w})}{\partial {\bf w}} &= \frac{\partial}{\partial {\bf w}} \left[ \sum_{i = 1}^{N} - \left[ y_i \log(\mu_i) + (1 - y_i) \log(1 - \mu_i) \right] \right] \\[2ex] &= \sum_{i = 1}^{N} \left[ -y_i \times \frac{1}{\mu({\bf x_i})} + (1 - y_i) \times \frac{1}{1 - \mu({\bf x_i})} \right] \frac{\partial}{\partial {\bf w}} \mu({\bf x_i}) \\[2ex] &= \sum_{i = 1}^{N} \left[ -y_i \times \frac{1}{\mu({\bf x_i})} + (1 - y_i) \times \frac{1}{1 - \mu({\bf x_i})} \right] \mu({\bf x_i})(1 - \mu({\bf x_i})){\bf x_i} \\[2ex] &= \sum_{i = 1}^{N} \left[ -y_i \times \left[ 1 - \mu({\bf x_i}) \right] + (1 - y_i) \mu({\bf x_i}) \right] {\bf x_i} \\[2ex] &= \sum_{i = 1}^{N} \left[ -y_i + \mu({\bf x_i}) \right] {\bf x_i} \\[2ex] &= \sum_{i = 1}^{N} \left[ \mu({\bf x_i}) - y_i \right] {\bf x_i} \end{array}$
二阶Hessian矩阵

$\begin{array}{ll} H (w) & = \frac{\partial}{\partial w} [g (w)^{T}] \\ = \sum_{i = 1}^{N} (\frac{\partial}{\partial w} μ_{i}) x_{i}^{T} \\ = \sum_{i = 1}^{N} μ_{i} (1 - μ_{i}) x_{i} x_{i}^{T} \\ = X^{T} d i a g (μ_{i} (1 - μ_{i})) X^{T} \\ = X^{T} S X & 正定矩阵，凸优化 \end{array}$ $\begin{array}{ll} {\bf H(w)} &= \frac{\partial}{\partial {\bf w}} \left[ {\bf g(w)^T} \right] \\[2ex] &= \sum_{i = 1}^{N} \left( \frac{\partial}{\partial {\bf w}} \mu_i \right) {\bf x}_i^T \\[2ex] &= \sum_{i = 1}^{N} \mu_i (1 - \mu_i) {\bf x_ix_i^T} \\[2ex] &= {\bf X^T} diag(\mu_i (1 - \mu_i)) {\bf X^T} \\[2ex] &= {\bf X^T S X} & \text{正定矩阵，凸优化} \end{array}$
牛顿法迭代机制

$w^{t + 1} = w^{y} - H^{- 1} (w^{t}) g (w^{t})$ $\bf w^{t + 1} = w^y - H^{-1}(w^t)g(w^t)$

也称二阶梯度下降法，移动方向为：

$d = - (H (w^{t}))^{- 1} g (w^{T})$ $\bf d = -(H(w^t))^{-1}g(w^T)$
拟牛顿法

基本思想是：不用二阶偏导数而构造出可以近似Hessian矩阵(或Hessian矩阵的逆矩阵)的正定对称矩阵，进而再逐步优化目标函数。

AI（003） - 笔记 - 第二周 - Logistic回归基本原理