LOGISTIC REGRESSION

Linear regression & Logistic regression

The purpose of linear regression is to predict a continuous variable $y$ with input $x$ .
Logistic regression mainly means to classify different categories with input $x$ .

The principle of logistic regression

LIke linear regression, there is also a predictive function $h_\theta(x)$ (called classification function, linear or non-linear function) in logistic regression.

first, compute the predictive function
second, call the sigmoid function
finally, get the loss function and optimize parameters

loss function

$J(\theta)=\frac{1}{N}\sum_{i=1}^NCost(h_\theta(x),y)$

optimization
1. batch gradient descent
  $\theta=\theta+\alpha*\frac{1}{N}\sum_{i=1}^N(y_i-g(X_\theta))$
  (repeat until convergence)
2. stochastic gradient descent
  for i to N:
  $\theta=\theta+\alpha*(y_i-h_\theta(x^{(i)}))x_j^{(i)}$
  (repeat until convergence)

Regularization

$J(\theta)=\frac{1}{N}\sum_{i=1}^Ny^{(i)}logh_\theta(x^{(i)})+(1-y^{(i)})log(1-h_\theta(x^{(i)}))+\frac{\lambda}{2N}\sum_{j=1}^N\theta_j^2$

Model evaluation index

receiver operating characteristic curve(roc)
the horizontal axis: false positive rate(fpr)
the vertical axis: true positive rate(tpr)
every point corresponds to a thresthold
area under curve(auc)
auc is related to roc

在这里插入图片描述

Advantages

easy to compute, understand and implement
efficient in time and memory
robustness is good for small noise in data

Disadvantages

underfitting happens easily
classification accuracy is not high
not work well for a big feature space

Sample imbalance issue

The logistic regression model ignores the feature of small sample category.

solutions

oversampling, undersampling and combined sampling
weight adjustment
kernel function correction
model correction

sklearn parameters

sklearn.linear_model.LogisticRegression(penalty='l2', dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight=None, random_state=None, solver='liblinear', max_iter=100, multi_class='ovr', verbose=0, warm_start=False, n_jobs=1)

regularization parameter: penalty(‘l1’ or ‘l2’)
optimization: solver(‘liblinear’ ‘lbfgs’ ‘newton-cg’ ‘sag’)
classification: multi_class(‘ovr’ or ‘multinomial’)
class weight: class_weight
sample weight: sample_weight