Machine learning linear regression (six): Linear regression from entry to the master - the normal equation optimization and gradient descent optimization method

Introduction to Linear Regression

Linear regression scenarios

• Prices forecast

• Sales Prediction

• Loan amount forecast

  For example:

What is the linear regression

Definitions and formulas

Linear Regression (Linear regression) using the regression equation (function) for between the independent variables (characteristic value) and the dependent variable (target value) of one or more relationships for modeling an analytical way.

• Features: with only one independent variable is called a single-variable regression, more than one independent variable is called multiple regression case

• Examples denoted by linear regression matrix

So how to understand it? Let's look at a few examples

• Final grade: 0.7 × test scores + 0.3 × normal results
• House prices from the central area = 0.02 × + 0.04 × city nitric oxide concentration + (-0.12 × average rate from the housing) + 0.254 × urban crime

Two examples above, we see the establishment of a relationship between the characteristic value and the target value, this relationship can be understood as a linear model .

Analysis of the relationship between characteristics and objectives of the linear regression

There are two main linear regression model which, ** one is a linear relationship, the other is a non-linear relationship. ** Here we can only draw a plane to be better understood, the features are for example a single or two sub-features.

• Linear relationship

  • Univariate linear relationship:

  

  • Multivariate linear relationship

    

NOTE: Relation between the target characteristics and a linear relationship, or wherein the target value exhibits two planar relationship

The higher dimensions we do not have to think, you can remember this relationship

• Non-linear relationship

  

Comment: Why is this relationship? what is the reason?

If the non-linear relationship, the regression equation can be understood as: w 1 x 1 + w 2 x 2 ^{2 + w3x3} 2

Preliminary using linear regression api

Linear regression API

• sklearn.linear_model.LinearRegression()
  • LinearRegression.coef_：回归系数

举例

步骤分析

• 1.获取数据集
• 2.数据基本处理（该案例中省略）
• 3.特征工程（该案例中省略）
• 4.机器学习
• 5.模型评估（该案例中省略）

代码过程

• 导入模块

from sklearn.linear_model import LinearRegression

• 构造数据集

x = [[80, 86],
[82, 80],
[85, 78],
[90, 90],
[86, 82],
[82, 90],
[78, 80],
[92, 94]]
y = [84.2, 80.6, 80.1, 90, 83.2, 87.6, 79.4, 93.4]

• 机器学习-- 模型训练

# 实例化API
estimator = LinearRegression()
# 使用fit方法进行训练
estimator.fit(x,y)

estimator.coef_

estimator.predict([[100, 80]])

数学:求导

常见函数的导数

导数的四则运算

练习

• 3.1 y = x^3-2x2+sinx,求f`(x)

• 3.2 y=ln(sinx), 求dy/dx

矩阵（向量）求导 [了解]

线性回归的损失和优化

假设刚才的房子例子，真实的数据之间存在这样的关系

真实关系：真实房子价格 = 0.02×中心区域的距离 + 0.04×城市一氧化氮浓度 + (-0.12×自住房平均房价) + 0.254×城镇犯罪率

那么现在呢，我们随意指定一个关系（猜测）

随机指定关系：预测房子价格 = 0.25×中心区域的距离 + 0.14×城市一氧化氮浓度 + 0.42×自住房平均房价 + 0.34×城镇犯罪率

请问这样的话，会发生什么？真实结果与我们预测的结果之间是不是存在一定的误差呢？类似这样样子

既然存在这个误差，那我们就将这个误差给衡量出来

损失函数

总损失定义为：

• yi为第i个训练样本的真实值
• h(xi)为第i个训练样本特征值组合预测函数
• 又称最小二乘法

如何去减少这个损失，使我们预测的更加准确些？既然存在了这个损失，我们一直说机器学习有自动学习的功能，在线性回归这里更是能够体现。这里可以通过一些优化方法去优化（其实是数学当中的求导功能）回归的总损失！！！

优化算法

如何去求模型当中的W，使得损失最小？（目的是找到最小损失对应的W值）

线性回归经常使用的两种优化算法

正规方程

什么是正规方程

理解：X为特征值矩阵，y为目标值矩阵。直接求到最好的结果

缺点：当特征过多过复杂时，求解速度太慢并且得不到结果

正规方程求解举例

以下表示数据为例：

即：

运用正规方程方法求解参数：

正规方程的推导

• 推导方式一：

把该损失函数转换成矩阵写法：

其中y是真实值矩阵，X是特征值矩阵，w是权重矩阵

对其求解关于w的最小值，起止y,X 均已知二次函数直接求导，导数为零的位置，即为最小值。

求导：

Note: (1) to the formula (2) derivation, X is a matrix of m rows and n columns, does not guarantee that there is an inverse matrix, but the right to take it into a matrix XT, to ensure that there is an inverse matrix .

Formula (5) to Formula (6) derivation process, and the similar.

• Derivation of the two [expand]:

https://www.jianshu.com/p/2b6633bd4d47