Data source:Boston area housing price prediction complete data set (CSV format)_weixin_51454889's blog-CSDN blog
References: "R Language Practical Combat"
(Actually, I don’t know much about the various checks and operations during actual regression, so the content of this article is very confusing. It mainly briefly introduces the method and its R language code. It is similar to one of my personal class notes, but I have my own knowledge reserve. It is limited, so it is inevitable that there will be many errors. Everyone is welcome to comment and discuss)
Table of contents
1.1 Introduction to each attribute
2.1 Gaussian Markov assumption: G-M assumption
2.2 Basic model: OLS estimation
3. Abnormal point inspection + impact analysis
3.2 High leverage value points
3.3 Strong influence points (impact analysis)
4.2.1 Gradual return of AIC standards
4.2.2 Fully Autoset Regression
4.2.3LASSO
5. Regression diagnosis (residual test)
5.2. Independence test (autocorrelation)*
5.4. Homoskedasticity test (heteroscedasticity)*
c. Weighted least squares method (special generalized least squares)
1. Introduction to data sets
1.1 Introduction to each attribute
1.2 Descriptive Statistics
>stat.desc(df)
library(pastecs)#描述性统计
des_stat<-stat.desc(df)
#输出为csv文件
write.table(des_stat,"描述性统计.csv",sep=",")
#sep：分隔符，默认为空格，则所有数据会在一个单元格里，因此要使用sep","
#row.names：是否导出行序号，默认为TRUE，也就是导出行序号
#col.names：是否导出列名，默认为TRUE，也就是导出列名
#quote：字符串是否使用引号表示，默认为TRUE，也就是使用引号表示
2. Basic return
2.1 Gaussian Markov assumption: G-M assumption
Only when the G-M assumption holds true, the estimation result of OLS is the best linear unbiased estimate (BLUE: linearity, unbiasedness, minimum variance, consistent and effective. (There may be an error in the above figure, the G-M assumption should actually be the top three hypothesis(?))
2.2 Basic model: OLS estimation
>lm(y~x1+x2+……)
fit<-lm(Murder~Population+Illiteracy+Income+Frost,data=states)
summary(fit)
2.3GLS estimate
(GLS estimation is not a benchmark regression, but it will be frequently used later, so it is introduced here)
Scope of application: Usually when the error term does not meet the "spherical disturbance term assumption" (ie, the homoskedasticity assumption and no autocorrelation assumption in the G-M assumption), GLS estimation can be used.
>GLS principle: (generalized least squares method)
· glm(y~x1+x2+……)
fit_gls=glm(Murder~Population+Illiteracy+Income+Frost,data=states)
summary(fit_gls)3. Abnormal point inspection + impact analysis
3.1 Outliers
Observations where the model predicts poorly (usually has large positive or negative residuals)
Judgment method: 1. The Q-Q plot residual falls outside the confidence interval; 2. Rough judgment: the absolute value of the studentized residual is greater than 2.
· outlierTest(fit): This function only tests the point with the largest standardized residual, and the others still need further testing)
· influencePlot(fit): The ordinate provides studentized residuals for judgment.
3.2 High leverage value points
Refers to the unsociable point in x
Judgment method: hat statistics. If the hat value of an observation point is greater than 2 to 3 times the average hat value, it is considered a high leverage point;
· hatvalues(fit)
hat.plot <- function(fit) {
p <- length(coefficients(fit))
n <- length(fitted(fit))
#hatvalues(fit)是计算fit中每个因变量的帽子统计量
plot(hatvalues(fit),main = "Index Plot of Hat Values")
abline(h=c(2,3)*p/n,col="red",lty=2)
identify(1:n,hatvalues(fit),names(hatvalues(fit)))
}
hat.plot(fit)
The test results show: 365, 380, 405, 410, and 418 are high leverage value points
3.3 Strong influence points (impact analysis)
(Impact analysis actually belongs to the content of regression diagnosis, and is placed here to facilitate comparative analysis of three abnormal points)
Strong influence point: a point that has a huge impact on the parameter estimates of the model and causes a proportional imbalance;
Judgment criterion: cook distance. Generally, if the Cook'sD value is greater than 4/(n-k-1), it is considered a strong influence point;
· cook.distance() can get the cook distance
cutoff <- 4/(nrow(df)-length(fit$coefficients)-2)
plot(fit, which = 4, cook.levels = cutoff)
abline(h=cutoff,lty=2,col="red")
We can see that 364, 368, and 372 are automatically identified as strong influence points.
3.4 Overall detection
· influencePlot()
influencePlot(fit,id.method="identify",main="Influence Plot",
sub="Circle size is proportional to Cook's distance")
In the picture of the output result: The ordinate represents the studentized residual (outlier point), the abscissa represents the hat statistic (high leverage value point), and the circle of the indicator represents Degree of influence (strong influence point). So the ordinate exceeds +2 or is less than can be considered as outliers, and the horizontal axis exceeds 0.2 or The sample point a> is a high leverage value, and the size of the circle is proportional to the impact. Points with large circles are likely to be strong influence points that have a disproportionate impact on the estimates of model parameters. 0.3
The results of the abnormal points automatically identified by this function are as shown in the table above.
3.5 Delete abnormal points. Delete the above abnormal points before proceeding with subsequent operations.
4. Basic model inspection
4.1 Multicollinearity
4.1.1 Inspection method
1. Intuitive judgment of correlation coefficient matrix (rough): cor()
round(cor(states),3) #保留三位数
2. Characteristic root method: If X'X has a very small characteristic root, it is considered that there is complex collinearity.
3. Variance expansion factor method
It is generally believed that VIF>10 has serious multicollinearity, while sqrt(VIF)>2 is considered to have multiple common linear problems;
· vif()
#判断多重共线性
vif_judge<-function(fit0){
vif(fit0)
sqrt(vif(fit0))>2
}
vif_judge(fit)
The test results of this experimental data show that it can be considered that there is no serious multicollinearity;
4.1.2 Ridge regression
Reference article:Machine learning--Linear regression 2 (collinearity problem, ridge regression, lasso algorithm)_zsffuture's blog-CSDN blog
Theoretical principle: (Essential meaning) A penalty term is imposed on X’X so that it is no longer infinitely close to 0. When k=0, it is ordinary OLS.
It is unknown how much lambda should be, so it is necessary to iterate and select an appropriate lambd to make the results of each estimate stabilize, that is, find a minimum lambda, and the ordinate corresponding to each parameter in the ridge trace diagram will no longer change significantly.
Ridge trace diagram: ordinate - estimated beta value, abscissa - lambda;
library(MASS)
#岭回归
gr<-lm.ridge(MEDV~ CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PIRATIO+B+LSTAT,
data=df1,
lambda=seq(0,100,0.01))
#绘制岭迹图
#法一：plot(gr,data =df1,lambda = seq(0,100,0.01))
#法二：
matplot(gr$lambda,t(gr$coef),type='l',
xlab=expression(lambda),
ylab=expression(hat(beta)),
main ='Ridge trace Map')
#选出合适的lambda
select(gr)
The output ridge trace plot is as follows: Obviously the data used in this question is not very multicollinear. When lambda is small, its change has almost no impact on the parameter estimate (if there is severe multicollinearity lambda=0, the absolute value of the estimated parameter should be is very large and is sensitive to changes in lambda. It gradually decreases and then stabilizes as it changes). So the data and model (1) in this example are not suitable for ridge regression.
Other ways to solve multicollinearity problems include principal component analysis (to be added) and partial least squares (to be added)
4.2 Variable selection
Traditional judgment accuracy tests: AIC criterion, BIC criterion, Cp criterion, modified R^2, RMSq, etc.
| method | Thought | shortcoming |
| optimal subset regression | Similar to the idea of exhaustive method. 1. Given the specified subset size m (m=1...M); 2. Screen out the m variables with the best results under each given subset size; 3. Finally, use external data or select appropriate criteria to select the optimal model from these M models. |
In fact, it is similar to the exhaustive method. When there are too many variables, it is very difficult to execute. Generally, when the number of variables exceeds 30, this method is almost ineffective. |
| K-fold CV method | 1. The original data is randomly divided into K folds, and the subset size m is specified (m=1...M), 2. The first fold is used as the validation set, and then the model is fitted on the remaining k-1 folds. The mean square error MSE1 is calculated from the observations of the retained folds. 3. Repeat the above steps k times, each time using different folds as the verification set. The entire process will obtain k test error estimates MSE1, MSE2,..., MSEk. These values are then averaged as the prediction error corresponding to each m. 4. Compare the average test error and finally select the optimal model; |
The essence is optimal subset regression, which is a further expansion in the final step of selecting the optimal model from M models. |
| step by step regression |
Choose the best at every step. 1. Do univariate regression first. Select the single variable with the best fitting effect; 2. Given the subset size m (m=1...M). New variables are gradually added on the basis of single variables, and each time the variable with the best fit is selected, and then the third variable is added, and so on, until there are m variables; 3. Use external data or select appropriate criteria to select the optimal model from these M models; |
Every step is pursued to the "extreme" but the result is not necessarily optimal. |
| stepwise regression backwards | The idea is similar to forward stepwise regression. But on the basis of the full model, one variable is reduced at a time. |
|
| forward shard regression | …… |
4.2.1 Gradual return of AIC standards
Generally speaking, the smaller the AIC and BIC, the better the fitting effect;
stepAIC(fit1,direction="backward")4.2.2 Fully Autoset Regression
Idea: All possible models in the fully autoset regression method will be tested, and the analysis can choose to display all possible results. Select the model with the largest modified resolvability coefficient.
library(leaps)
leaps<-regsubsets(MEDV~ CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PIRATIO+B+LSTAT,
data=df1,nbest=14)
#nbset代表展示n个不同子集大小的最佳模型
plot(leaps,scale="adjr2")
The full autoset regression results show that CRIM, INDUS, AGE, RAD, and TAX are eliminated, and the best model established is MEDV~ZN+CHAS NOX+RM+DIS+PIRATIO+B+LSTAT. At this time, the modified determination coefficient reaches The maximum is about 0.78. At the same time, several indicators with large VIF values are deleted, which reduces the impact of complex collinearity.
The regression results of the new model are shown in the figure below: At this time, the p values of the regression coefficients of all parameters are significant.
4.2.3 LASSO
The traditional AIC and BIC criteria mainly use discrete methods to screen variables, which have certain limitations. Currently, what is more popular in recent years is variable selection based on LASSO, which can overcome certain collinearity effects.
is a "shrinking method" that constructs a first-order absolute value penalty function (corresponding to ridge regression, it constructs a second-order penalty term of the L2 paradigm), and obtains a sparse solution to determine Select variable;
s is the adjustment parameter to be set. When s=0, the complexity is the lowest and contains only one constant term. When the value of s meets certain conditions, the complexity is the highest, and the LASSO solution degenerates into a least squares estimate;
The geometric meaning of LASSO: (Intuitive explanation of two-dimensional geometry)
The left side of the figure below is the geometric interpretation of LASSO, and the right figure is the geometric interpretation of ridge regression;
Under the LASSO constraint, you can see that as the ellipse grows, the ellipse and the blue part may intersect with the vertex on the coordinate system first. If it is on a high-dimensional coordinate family, there may be many variables at this vertex whose coefficients tend to 0. , so LASSO can be used to directly implement coefficient screening, but in the case of ridge regression, it is almost impossible to be tangent to the coordinate axis, so as to find the point where the coefficient tends to 0, so variable screening cannot be implemented.
(R language code: to be added)
library(lars)
object<-lars(x,y)
plot(object) #绘出LASSO回归的结果图，选择合适的变量
5. Regression diagnosis (residual test)
OLS regression diagnosisPurpose: 1. Diagnose the basic assumptions of OLS regression statistics on residuals (mainly check the normality of residuals, Independence, linearity, homoskedasticity) 2. Impact analysis: Explore whether there are strong influence points (see Section 3 for details)
Reference article:Linear regression model (least squares model) diagnosis--R language_Xiaobai 15138's blog-CSDN blog
(Almost the regression diagnosis part of "R Language Practice") (Because there are too many variables in this question, multicollinearity testing and variable selection were first performed, and currently only some independent variables remain)
Basic method:
fit<-lm(MEDV~ CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PIRATIO+B+LSTAT,
data=df)
par(mfrow=c(2,2))
plot(fit)
#自动绘制出：残差拟合图、标准化残差Q-Q图、位置尺度图、残差与杠杆图
The above four figures are: residual fitting plot (upper left - linear), standardized residual Q-Q plot (upper right - normality), position scale plot (lower left - homoskedasticity), residual and leverage plot (lower right - outlier problem);
From the residual fitting plot (upper left), it can be roughly seen that the residuals and the fitted values have a certain relationship, which proves that some data need to be transformed in the model; the standardized residual Q-Q plot (upper right) is almost approximately near the diagonal. The distribution can be roughly considered to comply with the assumption of normality;
Note: This method is relatively rough overall, and more detailed inspection can be carried out according to the following method.
5.1. Normality test:
Testing method:
a. Normal Q-Q plot of studentized residuals (including confidence interval)
dev.new()
qqPlot(fit,labels=row.names(df),id.method="identify",simulate=TRUE,main='Q-Q Plot')
#simulate=TRUE：自动生成95%置信区间
Automatically identify two unusual situations, 367 and 370.
b. Studentized residuals: draw the studentized residual histogram fitting curve and compare it with the standard normal distribution curve
residplot <- function(fit,nbreaks=10) {
#生成学生化残差
z <- rstudent(fit)
#绘制学生化残差的直方图
#参数freq=FALSE使得直方图生成的是频率图而不是频数图
hist(z,breaks=nbreaks,freq=FALSE,
xlab="Studentized Residual",
main="Distribution of Errors")
#绘制轴须图
rug(jitter(z),col="brown")
#curve函数常用于绘制函数对应的曲线，确定函数的表达式，以及对应的需要展示的起始坐标和终止坐标，curve函数就会自动化的绘制在该区间的函数图像
#如果参数add=TRUE,这时图像将在一个已经存在的图像上生成，这种情况下就可以不指定起始和终止坐标
curve(dnorm(x,mean=mean(z),sd=sd(z)),add=TRUE,col="blue",lwd=2)
lines(density(z)$x,density(z)$y,col="red",lwd=2,lty=2)
legend("topright",legend=c("Normal Curve","Kernel Density Curve"),
lty=1:2,col=c("blue","red"),cex=.7)
}
residplot(fit)
#————————————————
#版权声明：本文为CSDN博主「小白15138」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出
#处链接及本声明。
#原文链接：https://blog.csdn.net/weixin_42712867/article/details/998154495.2. Independence test (autocorrelation)*
There is autocorrelation between the error terms of the model.
>Autocorrelation test method:
a. Observe the residual plot; b. DW test (only first-order autocorrelation can be tested, the null hypothesis is pho=0)
· durbinWatsonTest(fit)
Obviously the p value is significant and the null hypothesis should be rejected. It is considered that the error terms are not independent of each other and have autocorrelation;
Correction method: generalized least squares method; iterative method, etc.
5.3.Linearity:
Use the component residual plot/bias residual plot to check whether it meets the settings; the residual is the vertical axis and the horizontal axis of the fitted value; there is no theoretical relationship between the two. If there is a relationship, you may need to add a quadratic value to the model. Items are improved.
· crPlots(fit)
It can be seen from the above figure that there may be a quadratic correlation between RM and MEDV, there may be a reciprocal correlation between LSTAT and MEDV, and the correlation between other variables and MEDV is not obvious. Therefore, the regression model was re-established and two terms RM^2 and 1/LSTAT were added to the original model to obtain the following model:
5.4. Homoskedasticity test (heteroscedasticity)*
If the homoscedasticity assumption is violated, then var(ei)=sigmai^2 changes as i changes, and is no longer an unchanged constant.
>Heteroskedasticity test method:
1. Draw a scatter plot for observation
2. G-Q test: Sort the sample data from small to large (when it is multi-dimensional, the principal component method can be used to sort high-dimensional data), divide it into three parts and remove the middle part, assuming that high and low observation values have homoskedasticity. Assume that their variances are sigma_l^2 and sigma_h^2 respectively
>Heteroskedasticity correction method: a. Take logarithms on both sides to alleviate; b. Box-Cox transformation; c. Weighted least squares method
b.box-cox transformation
Reference article:BOX-COX transformation (R language)_big white sheep_Aries' blog-CSDN blog_boxcox transformation r language
#解决方案一：box-cox变换
library(MASS)
bc<-boxcox(MEDV~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT,data=df1,lambda=seq(-2,2,0.01))
# λ的取值为区间[-2,2]上步长为0.01的值，bc中保存了λ的值及其对应的对数似然函数值
lambda<-bc$x[which.max(bc$y)] # 将使对数似然函数值达到最大的λ复制给lambda
lambda
y_bc<-(df1$MEDV^lambda-1)/lambda # 计算变换后的y值
lm_bc<-lm(y_bc~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT,data=df1) # 使用变换后的y值建立回归方程
summary(lm_bc)c. Weighted least squares method (special generalized least squares)
The principle is: give larger weights to variables with smaller residual estimates, and give smaller weights to variables with larger residual estimates.
Reference article:R language for weighted least squares_R language and econometrics (3) Heteroskedasticity_weixin_39672443's blog-CSDN blog
· lm(y~x,weight=)
#加权最小二乘法
e<-summary(fit3)$resid
lm_weight<-lm(MEDV~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT+I(RM^2)+I(1/LSTAT),
df1,weight=1/abs(e))
summary(lm_weight)
ncvTest(lm_weight)
3.2 Comprehensive verification
>gvlma()
library(gvlma)
gvmodel<-gvlma(fit)
summary(gvmodel)6. Empirical part
#1. Basic preparation
library(car)
library(MASS)
library(pastecs)#描述性统计
des_stat<-stat.desc(df)
#输出为csv文件
write.table(des_stat,"F:/个人嘿嘿嘿/北师大BNU/研一上-课业资料/应用多元线性回归/hw02多元回归/描述性统计.csv",sep=",")
#一、数据准备
df<-read.csv('F:/个人嘿嘿嘿/北师大BNU/研一上-课业资料/应用多元线性回归/hw02多元回归/boston_housing_data.csv')
df<-df[2:506,1:14] #剔除标题行
dim(df)
#前13列是自变量，第14列是响应变量
#给行和列命名
colnames(df)<-c("CRIM","ZN","INDUS","CHAS","NOX","RM","AGE",
"DIS","RAD","TAX","PIRATIO","B","LSTAT","MEDV")
rownames(df)<-seq(1:505)#2. Basic OLS
fit<-lm(MEDV~ CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PIRATIO+B+LSTAT,data=df)
summary(fit)#3. Preliminary inspection of the model
multicollinearity
#3.1诊断多重共线性
vif_judge<-function(fit_test){
print(vif(fit_test))
sqrt(vif(fit_test))>2
}
vif_judge(fit1)
#岭回归
library(MASS)
gr<-lm.ridge(MEDV~ CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PIRATIO+B+LSTAT,
data=df1,
lambda=seq(0,100,0.01))
#绘制岭迹图
plot(gr,data =df1,lambda = seq(0,100,0.01))
matplot(gr$lambda,t(gr$coef),type='l',
xlab=expression(lambda),
ylab=expression(hat(beta)),
main ='Ridge trace Map')
#选择出合适的lambda
select(gr)Variable selection: AIC stepwise regression method, fully autoset regression
#3.2变量选择
#逐步回归法stepwise method
#删除了CRIM和INDUS
stepAIC(fit1,direction="backward")
#全子集回归
#显示：删除CRIM INDUS AGE RAD TAX
library(leaps)
leaps<-regsubsets(MEDV~ CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PIRATIO+B+LSTAT,
data=df1,nbest=14)
plot(leaps,scale="adjr2")
#模型二：变量选择后OLS
fit2<-lm(MEDV~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT,
data=df1)
summary(fit2)#3.3异常点判断
influencePlot(fit,id.method="identify",main="Influence Plot",sub="Circle size is proportional to Cook's distance")
#3.3.1高杠杆值点
#365 380 405 410 418
hat.plot <- function(fit) {
p <- length(coefficients(fit))
n <- length(fitted(fit))
#hatvalues(fit)是计算fit中每个因变量的帽子统计量
plot(hatvalues(fit),main = "Index Plot of Hat Values")
abline(h=c(2,3)*p/n,col="red",lty=2)
identify(1:n,hatvalues(fit),names(hatvalues(fit)))
}
hat.plot(fit)
#3.3.2强影响点
#368 371 372 380 418
cutoff <- 4/(nrow(df)-length(fit$coefficients)-2)
plot(fit, which = 4, cook.levels = cutoff)
abline(h=cutoff,lty=2,col="red")
#剔除异常值
df1<-df[-c(364,365,368,371,372,380,405,410,418),]
dim(df1)
fit1<-lm(MEDV~ CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PIRATIO+B+LSTAT,
data=df1)#4. Regression diagnosis
#四、回归诊断
#粗糙的回归诊断
par(mfrow=c(2,2))
plot(fit2)
#4.1进一步：正态性检验
qqPlot(fit2,labels=row.names(df),id.method="identify",simulate=TRUE,main='Q-Q Plot')
#4.2进一步：独立性检验
durbinWatsonTest(fit2)#4.3进一步：线性检验
crPlots(fit2)
#针对线性的修正：
fit3<-lm(MEDV~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT+I(RM^2)+I(1/LSTAT),data=df1)
summary(fit3)
durbinWatsonTest(fit3)#4.4进一步：同方差检验
ncvTest(fit2)
#计分检验显著（p=0.32607),说明不拒绝同方差原假设
spreadLevelPlot(fit)
#4.4.1解决方案：加权最小二乘法->异方差
e<-summary(fit3)$resid
lm_weight<-
lm(MEDV~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT+I(RM^2)+I(1/LSTAT),df1,weight=1/abs(e))
summary(lm_weight)
ncvTest(lm_weight)box-cox transformation
#4.5box-cox变换
library(MASS)
bc<-boxcox(MEDV~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT,data=df1,lambda=seq(-2,2,0.01))
# λ的取值为区间[-2,2]上步长为0.01的值，bc中保存了λ的值及其对应的对数似然函数值
lambda<-bc$x[which.max(bc$y)] # 将使对数似然函数值达到最大的λ复制给lambda
lambda
y_bc<-(df1$MEDV^lambda-1)/lambda # 计算变换后的y值
lm_bc<-lm(y_bc~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT,data=df1) # 使用变换后的y值建立回归方程
summary(lm_bc)
#再次诊断
durbinWatsonTest(lm_bc)
ncvTest(lm_bc)GLS method
#4.6GLS
fit_gls<-lm(MEDV~ZN+CHAS+NOX+RM+DIS+PIRATIO+B+LSTAT,
data=df1)
summary(fit_gls)
durbinWatsonTest(fit_gls) #DW=1.0827 p=0 不互独
ncvTest(fit_gls) #p=0.00011125 异方差