Logistic regression and regularization

logistic regression

1. Used to solve classification problems (just named regression for historical reasons)

2. Binary classification

  • Definition: For input, the output value is not continuous, but two discrete values, eg:{0,1}
  • Method: Using linear regression, set the output predicted value greater than 0.5 to 1, and the output predicted value less than 0.5 to 0. (Currently not feasible, because the classification problem is not a linear function, so the Sigmoid Function (Sigmoid Function)/ logistic function)
  • Sigmoid Function / logistic function

          

          z>=0时g>=0.5,z<0时g<0.5; z-> -∞,g->0;  z-> +∞,g->1

        

         h(x) is the probability that the output value is 1:

          

  • To get discrete classifications, assume:

        ,So have:

         

  • Decision boundary (decision boundary): The line that separates the area of ​​y=0 and y=1 (corresponding to the above equation of θ'x=0) 
  • Cost function:

        

        Combine the above two equations:

       

        Re-vectorized representation:

        

         The drawing is:

                         

  • Gradient Descent (gradient decent):

           

         The partial derivative can be obtained: (same as the previous linear regression result)

         

         The vectorized representation is:

         

  • A better way to find θ than gradient descent:

   Conjugate gradient, BFGS, L-BFGS

        Use octave's internal function library to call these methods, steps:

    1. Write the cost function and its partial derivatives:
   function [jVal, gradient] = costFunction(theta)
     jVal = [...code to compute J(theta)...];
     gradient = [...code to compute derivative of J(theta)...];
   end 
2. Call the fminunc function, optimset is the parameter passed to the function
   options = optimset ('GradObj', 'on', 'MaxIter', 100);
   initialTheta = zeros(2,1);
   [optTheta, functionVal, exitFlag] = fminunc(@costFunction, initialTheta, options);

3. Multiclass classification:

  • The result is multiple categories, y={0,1,....n}
  • In this case, the problem can be divided into n+1 binary classification problems, select a class, treat all other classes as one class, use the binary classification, find each h(x), and then find the maximum value

        

 

regularization 

1. Regularization is used to solve overfitting problems (overfitting), suitable for linear regression and logistic regression problems.

    extend:

  • The h function is too simple: underfitting, high bias (underfitting, high bias)
  • The h function is too complicated: overfitting, high variance (overfitting, high variance)
  • Overfitting methods include: reducing parameters (manual parameter selection or using a specific parameter selection algorithm), regularization

2. The normalized cost function:

       (λ is the normalization parameter)

    Demo: https://www.desmos.com/calculator/1hexc8ntqp

    eg:

       h(x)= 

       If you want to reduce the influence of theta3, theta4 parameters, the cost function can be written as follows:

       

       The last two parameters are multiplied by a large number, so that in order to obtain the minimum value of the cost, the last two parameters are very small and approach 0, so that the two parameters in the h function are

       will be small, which is equivalent to eliminating the parameters, so the function will become flat and will not overfit

3. Regularized Linear Regression:

  • Gradient descent becomes: (additional λ term)

        

         Shift can also be written as:

         

         It can be seen that compared with no normalization, theta becomes the original (1-α*λ/m)<1 times in each iteration.

  • The non-iterative nominal equation is:

             ( L=(n+1)*(n+1) )

4. Regularized logistic regression:

  • The cost function is: (addition of λ terms)

          (note that there is no item 0)

  • Gradient descent: (same as linear regression regularization)

        

  

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=326001131&siteId=291194637