Machine Learning - locally weighted linear regression and linear regression

Machine Learning - Linear Regression

Code for this article are from the "real machine learning"

Classification algorithm first stop here, went on to say a regression algorithm

Linear Regression

Linear regression is relatively simple, not how to say, if the model can not remember about it on Baidu, listed here about the formula directly on the code

'''
Created on Jan 8, 2011

@author: Peter
'''
from numpy import *
#加载数据
def loadDataSet(fileName):      #general function to parse tab -delimited floats
    #attribute的个数
    numFeat = len(open(fileName).readline().split('\t')) - 1 #get number of fields 
    dataMat = []; labelMat = []
    fr = open(fileName)
    for line in fr.readlines():
        lineArr =[]
        curLine = line.strip().split('\t')
        for i in range(numFeat):
            lineArr.append(float(curLine[i]))
        #dataMat是一个二维矩阵，labelMat是一维的
        dataMat.append(lineArr)
        labelMat.append(float(curLine[-1]))
    return dataMat,labelMat
#就是，简单的线性回归
def standRegres(xArr,yArr):
    #这里没有出现为特征X加1列，因为它已经写在数据中了2333，要是数据中没有的话是要写的。
    xMat = mat(xArr); yMat = mat(yArr).T
    xTx = xMat.T*xMat#这个是X的转置乘以X
    if linalg.det(xTx) == 0.0:
        print("This matrix is singular, cannot do inverse")
        return
    #.I在numpy中是求逆矩阵
    ws = xTx.I * (xMat.T*yMat)
    return ws

One problem is that linear regression may occur underfitting phenomenon, because it is not required with the minimum mean square error of unbiased estimate, so if the model underfitting would be impossible to get the best prediction. So some method allows estimation cited as some deviations, thereby reducing the mean square error of prediction. One way is locally weighted linear regression

--- of course, is the source of "machine learning real."

Locally Weighted Linear Regression

This page book more dry goods, directly impress

Did not see, people are using a kernel function, and linear regression you the impression that silly sweet white is not the same! I know the book Why put it in the back of the bar SVM

Then the locally weighted linear regression is the story behind it? See the Internet a more reliable blog, summarized as follows:

Source: http: //cynhard.com/2018/07/13/ML-Locally-Weighted-Linear-Regression/

(Because of the bloggers use the plugin to display the formula, I shot a lazy 2333)

It can be seen, the so-called "weight" is not directly applied to the linear equation, but applied to the loss function, the function used to "deviation" of.

Key stroke: known samples closer the prediction sample, the greater the weight the weight; the larger k, the weight decreases with distance more slowly .

Locally weighted regression is based on the idea of ​​non-parametric learning algorithms, such selection characteristics better. Giving each a point near the predicted point at a certain weight, in it to be based on a function of wavelength ordinary linear regression fit to be accurate near the point while ignoring the contribution of those distant points, or nearly point large weights, weights far point a small value, k is the wavelength of parameters, controlling the rate of weight decrease with distance, the greater the faster the fall. The smaller and more precise fit small problems may occur too.

The main drawback of the algorithm is required for each forecast point, based on the entire data set to be re-simulate a linear regression model come out, so that the cost of the algorithm is extremely high. ----------------
Disclaimer: This article is CSDN blogger original article, "Beautiful sudden" follow CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source and link this statement.
Original link: https: //blog.csdn.net/tianse12/article/details/70161591

Locally weighted linear regression does not require "learning", belonging to the same properties and KNN, not even as the nature of this linear regression

code show as below:

'''
Created on Jan 8, 2011

@author: Peter
'''
from numpy import *
#局部加权线性回归，需要手动选择一个合适的k
#输入的是一个向量而不是矩阵，它一次只能处理一个预测！有点捞
def lwlr(testPoint,xArr,yArr,k=1.0):
    #x和y是训练集样本
    xMat = mat(xArr); yMat = mat(yArr).T
    #m是样本个数
    m = shape(xMat)[0]
    #创建一个对角单位矩阵用来保存权重w
    #使用矩阵而不是向量的原因是为了最后用矩阵乘法好算，公式里面也是这样写的
    weights = mat(eye((m)))
    for j in range(m):                      #next 2 lines create weights matrix
        #一个样本计算一遍，填到weight中
        diffMat = testPoint - xMat[j,:]     #
        weights[j,j] = exp(diffMat*diffMat.T/(-2.0*k**2))
    xTx = xMat.T * (weights * xMat)
    if linalg.det(xTx) == 0.0:
        print ("This matrix is singular, cannot do inverse")
        return
    ws = xTx.I * (xMat.T * (weights * yMat))
    return testPoint * ws
#对多个待回归点进行回归
def lwlrTest(testArr,xArr,yArr,k=1.0):  #loops over all the data points and applies lwlr to each one
    m = shape(testArr)[0]
    yHat = zeros(m)
    for i in range(m):
        yHat[i] = lwlr(testArr[i],xArr,yArr,k)
    return yHat
#输出x和y坐标，比lwlrTest更方便将结果画出来
def lwlrTestPlot(xArr,yArr,k=1.0):  #same thing as lwlrTest except it sorts X first
    yHat = zeros(shape(yArr))       #easier for plotting
    xCopy = mat(xArr)
    xCopy.sort(0)#按第一个特征排个序
    for i in range(shape(xArr)[0]):
        yHat[i] = lwlr(xCopy[i],xArr,yArr,k)
    return yHat,xCopy