Multiple Regression
When to use it
Null hypothesis
How it works
Using nominal variables in a multiple regression
Selecting variables in multiple regression
Assumptions
See the Handbook for information on these topics.
Example
The Maryland Biological Stream Survey example is shown in the “How to do the multiple regression” section.
Graphing the results
Similar tests
See the Handbook for information on these topics.
How to do multiple regression
Multiple correlation
Whenever you have a dataset with multiple numeric variables, it is a good idea to look at the correlations among these variables. One reason is that if you have a dependent variable, you can easily see which independent variables correlate with that dependent variable. A second reason is that if you will be constructing a multiple regression model, adding an independent variable that is strongly correlated with an independent variable already in the model is unlikely to improve the model much, and you may have good reason to chose one variable over another.
Finally, it is worthwhile to look at the distribution of the numeric variables. If the distributions differ greatly, using Kendall or Spearman correlations may be more appropriate. Also, if independent variables differ in distribution from the dependent variable, the independent variables may need to be transformed. In this example, Longnose, Acreage, Maxdepth, NO3, and SO4 are relatively log-normally distributed, while DO2 and Temp are relatively normal in distribution. It may be advisable in this case to transform these variable so that they all have similar distributions (not shown here).
With the corr.test function in the psych package, the “Correlation matrix” shows r-values and the “Probability values” table shows p-values. The PerformanceAnalytics plot shows r-values, with asterisks indicating significance, as well as a histogram of the individual variables. Either of these indicates that Longnose is significantly correlated with Acreage, Maxdepth, and NO3.
### --------------------------------------------------------------
### Multiple correlation and regression, stream survey example
### pp. 236–237
### --------------------------------------------------------------
Input = ("
Stream Longnose Acerage DO2 Maxdepth NO3 SO4 Temp
BASIN_RUN 13 2528 9.6 80 2.28 16.75 15.3
BEAR_BR 12 3333 8.5 83 5.34 7.74 19.4
BEAR_CR 54 19611 8.3 96 0.99 10.92 19.5
BEAVER_DAM_CR 19 3570 9.2 56 5.44 16.53 17
BEAVER_RUN 37 1722 8.1 43 5.66 5.91 19.3
BENNETT_CR 2 583 9.2 51 2.26 8.81 12.9
BIG_BR 72 4790 9.4 91 4.1 5.65 16.7
BIG_ELK_CR 164 35971 10.2 81 3.2 17.53 13.8
BIG_PIPE_CR 18 25440 7.5 120 3.53 8.2 13.7
BLUE_LICK_RUN 1 2217 8.5 46 1.2 10.85 14.3
BROAD_RUN 53 1971 11.9 56 3.25 11.12 22.2
BUFFALO_RUN 16 12620 8.3 37 0.61 18.87 16.8
BUSH_CR 32 19046 8.3 120 2.93 11.31 18
CABIN_JOHN_CR 21 8612 8.2 103 1.57 16.09 15
CARROLL_BR 23 3896 10.4 105 2.77 12.79 18.4
COLLIER_RUN 18 6298 8.6 42 0.26 17.63 18.2
CONOWINGO_CR 112 27350 8.5 65 6.95 14.94 24.1
DEAD_RUN 25 4145 8.7 51 0.34 44.93 23
DEEP_RUN 5 1175 7.7 57 1.3 21.68 21.8
DEER_CR 26 8297 9.9 60 5.26 6.36 19.1
DORSEY_RUN 8 7814 6.8 160 0.44 20.24 22.6
FALLS_RUN 15 1745 9.4 48 2.19 10.27 14.3
FISHING_CR 11 5046 7.6 109 0.73 7.1 19
FLINTSTONE_CR 11 18943 9.2 50 0.25 14.21 18.5
GREAT_SENECA_CR 87 8624 8.6 78 3.37 7.51 21.3
GREENE_BR 33 2225 9.1 41 2.3 9.72 20.5
GUNPOWDER_FALLS 22 12659 9.7 65 3.3 5.98 18
HAINES_BR 98 1967 8.6 50 7.71 26.44 16.8
HAWLINGS_R 1 1172 8.3 73 2.62 4.64 20.5
HAY_MEADOW_BR 5 639 9.5 26 3.53 4.46 20.1
HERRINGTON_RUN 1 7056 6.4 60 0.25 9.82 24.5
HOLLANDS_BR 38 1934 10.5 85 2.34 11.44 12
ISRAEL_CR 30 6260 9.5 133 2.41 13.77 21
LIBERTY_RES 12 424 8.3 62 3.49 5.82 20.2
LITTLE_ANTIETAM_CR 24 3488 9.3 44 2.11 13.37 24
LITTLE_BEAR_CR 6 3330 9.1 67 0.81 8.16 14.9
LITTLE_CONOCOCHEAGUE_CR 15 2227 6.8 54 0.33 7.6 24
LITTLE_DEER_CR 38 8115 9.6 110 3.4 9.22 20.5
LITTLE_FALLS 84 1600 10.2 56 3.54 5.69 19.5
LITTLE_GUNPOWDER_R 3 15305 9.7 85 2.6 6.96 17.5
LITTLE_HUNTING_CR 18 7121 9.5 58 0.51 7.41 16
LITTLE_PAINT_BR 63 5794 9.4 34 1.19 12.27 17.5
MAINSTEM_PATUXENT_R 239 8636 8.4 150 3.31 5.95 18.1
MEADOW_BR 234 4803 8.5 93 5.01 10.98 24.3
MILL_CR 6 1097 8.3 53 1.71 15.77 13.1
MORGAN_RUN 76 9765 9.3 130 4.38 5.74 16.9
MUDDY_BR 25 4266 8.9 68 2.05 12.77 17
MUDLICK_RUN 8 1507 7.4 51 0.84 16.3 21
NORTH_BR 23 3836 8.3 121 1.32 7.36 18.5
NORTH_BR_CASSELMAN_R 16 17419 7.4 48 0.29 2.5 18
NORTHWEST_BR 6 8735 8.2 63 1.56 13.22 20.8
NORTHWEST_BR_ANACOSTIA_R 100 22550 8.4 107 1.41 14.45 23
OWENS_CR 80 9961 8.6 79 1.02 9.07 21.8
PATAPSCO_R 28 4706 8.9 61 4.06 9.9 19.7
PINEY_BR 48 4011 8.3 52 4.7 5.38 18.9
PINEY_CR 18 6949 9.3 100 4.57 17.84 18.6
PINEY_RUN 36 11405 9.2 70 2.17 10.17 23.6
PRETTYBOY_BR 19 904 9.8 39 6.81 9.2 19.2
RED_RUN 32 3332 8.4 73 2.09 5.5 17.7
ROCK_CR 3 575 6.8 33 2.47 7.61 18
SAVAGE_R 106 29708 7.7 73 0.63 12.28 21.4
SECOND_MINE_BR 62 2511 10.2 60 4.17 10.75 17.7
SENECA_CR 23 18422 9.9 45 1.58 8.37 20.1
SOUTH_BR_CASSELMAN_R 2 6311 7.6 46 0.64 21.16 18.5
SOUTH_BR_PATAPSCO 26 1450 7.9 60 2.96 8.84 18.6
SOUTH_FORK_LINGANORE_CR 20 4106 10.0 96 2.62 5.45 15.4
TUSCARORA_CR 38 10274 9.3 90 5.45 24.76 15
WATTS_BR 19 510 6.7 82 5.25 14.19 26.5
")
Data = read.table(textConnection(Input),header=TRUE)
### Create a new data frame with only the numeric variables.
### This is required for corr.test and chart.Correlation
library(dplyr)
Data.num =
select(Data,
Longnose,
Acerage,
DO2,
Maxdepth,
NO3,
SO4,
Temp)
library(FSA)
headtail(Data.num)
Longnose Acerage DO2 Maxdepth NO3 SO4 Temp
1 13 2528 9.6 80 2.28 16.75 15.3
2 12 3333 8.5 83 5.34 7.74 19.4
3 54 19611 8.3 96 0.99 10.92 19.5
66 20 4106 10.0 96 2.62 5.45 15.4
67 38 10274 9.3 90 5.45 24.76 15.0
68 19 510 6.7 82 5.25 14.19 26.5
library(psych)
corr.test(Data.num,
use = "pairwise",
method="pearson",
adjust="none", # Can adjust p-values; see ?p.adjust for options
alpha=.05)
Correlation matrix
Longnose Acerage DO2 Maxdepth NO3 SO4 Temp
Longnose 1.00 0.35 0.14 0.30 0.31 -0.02 0.14
Acerage 0.35 1.00 -0.02 0.26 -0.10 0.05 0.00
DO2 0.14 -0.02 1.00 -0.06 0.27 -0.07 -0.32
Maxdepth 0.30 0.26 -0.06 1.00 0.04 -0.05 0.00
NO3 0.31 -0.10 0.27 0.04 1.00 -0.09 0.00
SO4 -0.02 0.05 -0.07 -0.05 -0.09 1.00 0.08
Temp 0.14 0.00 -0.32 0.00 0.00 0.08 1.00
Sample Size
Probability values (Entries above the diagonal are adjusted for multiple tests.)
Longnose Acerage DO2 Maxdepth NO3 SO4 Temp
Longnose 0.00 0.00 0.27 0.01 0.01 0.89 0.26
Acerage 0.00 0.00 0.86 0.03 0.42 0.69 0.98
DO2 0.27 0.86 0.00 0.64 0.02 0.56 0.01
Maxdepth 0.01 0.03 0.64 0.00 0.77 0.69 0.97
NO3 0.01 0.42 0.02 0.77 0.00 0.48 0.99
SO4 0.89 0.69 0.56 0.69 0.48 0.00 0.52
Temp 0.26 0.98 0.01 0.97 0.99 0.52 0.00
pairs(data=Data,
~ Longnose + Acerage + DO2 + Maxdepth + NO3 + SO4 + Temp)
library(PerformanceAnalytics)
chart.Correlation(Data.num,
method="pearson",
histogram=TRUE,
pch=16)
Multiple regression
Model selection using the step function
The step function has options to add terms to a model (direction="forward"), remove terms from a model (direction="backward"), or to use a process that both adds and removes terms (direction="both"). It uses AIC (Akaike information criterion) as a selection criterion. You can use the option k = log(n) to use BIC instead.
You can add the test="F" option to see the p-value for adding or removing terms, but the test will still follow the AIC statistic. If you use this, however, note that a significant p-value essentially argues for the term being included in the model, whether it’s its addition or its removal that’s being considered.
A full model and a null are defined, and then the function will follow a procedure to find the model with the lowest AIC. The final model is shown at the end of the output, with the Call: indication, and lists the coefficients for that model.
Stepwise procedure
model.null = lm(Longnose ~ 1,
data=Data)
model.full = lm(Longnose ~ Acerage + DO2 + Maxdepth + NO3 + SO4 + Temp,
data=Data)
step(model.null,
scope = list(upper=model.full),
direction="both",
data=Data)
Longnose ~ 1
Df Sum of Sq RSS AIC
+ Acerage 1 17989.6 131841 518.75
+ NO3 1 14327.5 135503 520.61
+ Maxdepth 1 13936.1 135894 520.81
<none> 149831 525.45
+ Temp 1 2931.0 146899 526.10
+ DO2 1 2777.7 147053 526.17
+ SO4 1 45.3 149785 527.43
.
.
< snip... more steps >
.
.
Longnose ~ Acerage + NO3 + Maxdepth
Df Sum of Sq RSS AIC
<none> 107904 509.13
+ Temp 1 2948.0 104956 509.24
+ DO2 1 669.6 107234 510.70
- Maxdepth 1 6058.4 113962 510.84
+ SO4 1 5.9 107898 511.12
- Acerage 1 14652.0 122556 515.78
- NO3 1 16489.3 124393 516.80
Call:
lm(formula = Longnose ~ Acerage + NO3 + Maxdepth, data = Data)
Coefficients:
(Intercept) Acerage NO3 Maxdepth
-23.829067 0.001988 8.673044 0.336605
Define final model
model.final = lm(Longnose ~ Acerage + Maxdepth + NO3,
data=Data)
summary(model.final) # Show coefficients, R-squared, and overall p-value
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2.383e+01 1.527e+01 -1.560 0.12367
Acerage 1.988e-03 6.742e-04 2.948 0.00446 **
Maxdepth 3.366e-01 1.776e-01 1.896 0.06253 .
NO3 8.673e+00 2.773e+00 3.127 0.00265 **
Multiple R-squared: 0.2798, Adjusted R-squared: 0.2461
F-statistic: 8.289 on 3 and 64 DF, p-value: 9.717e-05
Analysis of variance for individual terms
library(car)
Anova(model.final,
Type="II")
Anova Table (Type II tests)
Response: Longnose
Sum Sq Df F value Pr(>F)
Acerage 14652 1 8.6904 0.004461 **
Maxdepth 6058 1 3.5933 0.062529 .
NO3 16489 1 9.7802 0.002654 **
Residuals 107904 64
Simple plot of predicted values with 1-to-1 line
Data$predy = predict(model.final)
plot(predy ~ Longnose,
data=Data,
pch = 16,
xlab="Actual response value",
ylab="Predicted response value")
abline(0,1, col="blue", lwd=2)
Checking assumptions of the model
hist(residuals(model.final),
col="darkgray")
A histogram of residuals from a linear model. The distribution of these residuals should be approximately normal.
plot(fitted(model.final),
residuals(model.final)
)
A plot of residuals vs. predicted values. The residuals should be unbiased and homoscedastic. For an illustration of these properties, see this diagram by Steve Jost at DePaul University: condor.depaul.edu/sjost/it223/documents/resid-plots.gif.
### additional model checking plots with: plot(model.final)
Model fit criteria
Model fit criteria are available to decide which model is most appropriate. The step function uses AIC, or optionally BIC, but there are others. You don’t want to use multiple R-squared, because it will continue to improve as more terms are added into the model. Instead, you want to use a criterion that balances the improvement in explanatory power with not adding extraneous terms to the model. Adjusted R-squared is a modification of R-squared that includes this balance. Larger is better. AIC is based on information theory and measures this balance. AICc is an adjustment to AIC that is more appropriate for data sets with relatively fewer observations. BIC is similar to AIC, but penalizes more for additional terms in the model. Smaller is better for AIC, AICc, and BIC. There are differing opinions on which model fitting criteria is best to use, but if you have no opinion, I would recommend AICc for routine use.
Using the step procedure to automatically find an optimal model is an option, but some people caution against using an automated procedure because it might not hone in on the best model. Instead, you can look at the model fit criteria for competing models manually. There may be reasons why you wish to include or exclude some terms in the model, and it may be useful to look at different model selection criteria simultaneously.
In my compare.lm function below, Shapiro.W and Shapiro.p are results from the Shapiro–Wilks test for normality on the model residuals. A higher Shapiro W and a higher Shapiro p indicate that the residuals are more normally distributed. You should be aware, however, that any model with a high number of observation may yield a significant p-value (p < 0.05) for the Shapiro–Wilks test. It is best to investigate the residuals visually.
In the following example, we’ll look only at the terms that are significantly correlated with Longnose(Acreage, Maxdepth, and NO3), and then add in the other terms just to show the decrease in AICc by adding extra terms.
Note that AIC and BIC are calculated differently than in the step function.
model.1 = lm(Longnose ~ Acerage, data=Data)
model.2 = lm(Longnose ~ Maxdepth, data=Data)
model.3 = lm(Longnose ~ NO3, data=Data)
model.4 = lm(Longnose ~ Acerage + Maxdepth, data=Data)
model.5 = lm(Longnose ~ Acerage + NO3, data=Data)
model.6 = lm(Longnose ~ Maxdepth + NO3, data=Data)
model.7 = lm(Longnose ~ Acerage + Maxdepth + NO3, data=Data)
model.8 = lm(Longnose ~ Acerage + Maxdepth + NO3 + DO2, data=Data)
model.9 = lm(Longnose ~ Acerage + Maxdepth + NO3 + SO4, data=Data)
model.10 = lm(Longnose ~ Acerage + Maxdepth + NO3 + Temp, data=Data)
library(rcompanion)
compareLM(model.1, model.2, model.3, model.4, model.5, model.6,
model.7, model.8, model.9, model.10)
$Models
Formula
1 "Longnose ~ Acerage"
2 "Longnose ~ Maxdepth"
3 "Longnose ~ NO3"
4 "Longnose ~ Acerage + Maxdepth"
5 "Longnose ~ Acerage + NO3"
6 "Longnose ~ Maxdepth + NO3"
7 "Longnose ~ Acerage + Maxdepth + NO3"
8 "Longnose ~ Acerage + Maxdepth + NO3 + DO2"
9 "Longnose ~ Acerage + Maxdepth + NO3 + SO4"
10 "Longnose ~ Acerage + Maxdepth + NO3 + Temp"
$Fit.criteria
Rank Df.res AIC AICc BIC R.squared Adj.R.sq p.value Shapiro.W Shapiro.p
1 2 66 713.7 714.1 720.4 0.12010 0.10670 3.796e-03 0.7278 6.460e-10
2 2 66 715.8 716.2 722.4 0.09301 0.07927 1.144e-02 0.7923 2.115e-08
3 2 66 715.6 716.0 722.2 0.09562 0.08192 1.029e-02 0.7361 9.803e-10
4 3 65 711.8 712.4 720.6 0.16980 0.14420 2.365e-03 0.7934 2.250e-08
5 3 65 705.8 706.5 714.7 0.23940 0.21600 1.373e-04 0.7505 2.055e-09
6 3 65 710.8 711.4 719.6 0.18200 0.15690 1.458e-03 0.8149 8.405e-08
7 4 64 704.1 705.1 715.2 0.27980 0.24610 9.717e-05 0.8108 6.511e-08
8 5 63 705.7 707.1 719.0 0.28430 0.23890 2.643e-04 0.8041 4.283e-08
9 5 63 706.1 707.5 719.4 0.27990 0.23410 3.166e-04 0.8104 6.345e-08
10 5 63 704.2 705.6 717.5 0.29950 0.25500 1.409e-04 0.8225 1.371e-07
### Model 7 is the model which minimizes AICc, which is the same model
### chosen by the step function
Result = compare.lm(model.1, model.2, model.3, model.4, model.5, model.6,
model.7, model.8, model.9, model.10)
plot(Result$Fit.criteria$AICc,
xlab = "Model number",
ylab = "AICc")
lines(Result$Fit.criteria$AICc)
A plot of AICc (modified Akaike information criterion) of several models. Model 7 minimizes AICc, and is therefore chosen as the best model out of this set.
Comparing models with likelihood ratio test
It may also be helpful to compare models with the extra sum of squares test or likelihood ratio test to see if additional terms significantly reduce the error sum of squares.
One of the compared models should be nested within the other. That is, the one model should be the same as the other, except with additional terms. For example in the set of models below, it is appropriate to compare model.7 to model.4. Or to compare each of model.8, model.9, and model.10 to model.7.
For a single comparison, the anova function can be used for the Extra SS test, or lrtest in lmtest can be used for the likelihood ratio test. For multiple comparisons, the extraSS and lrt functions in the FSA package can be used. The extraSS function works only for lm and nls models, whereas the lrt function works on a wider range of model objects.
model.4 = lm(Longnose ~ Acerage + Maxdepth, data=Data)
model.7 = lm(Longnose ~ Acerage + Maxdepth + NO3, data=Data)
model.8 = lm(Longnose ~ Acerage + Maxdepth + NO3 + DO2, data=Data)
model.9 = lm(Longnose ~ Acerage + Maxdepth + NO3 + SO4, data=Data)
model.10 = lm(Longnose ~ Acerage + Maxdepth + NO3 + Temp, data=Data)
anova(model.7, model.4)
Analysis of Variance Table
Model 1: Longnose ~ Acerage + Maxdepth + NO3
Model 2: Longnose ~ Acerage + Maxdepth
Res.Df RSS Df Sum of Sq F Pr(>F)
1 64 107904
2 65 124393 -1 -16489 9.7802 0.002654 **
library(lmtest)
lrtest(model.7, model.4)
Likelihood ratio test
Model 1: Longnose ~ Acerage + Maxdepth + NO3
Model 2: Longnose ~ Acerage + Maxdepth
#Df LogLik Df Chisq Pr(>Chisq)
1 5 -347.05
2 4 -351.89 -1 9.6701 0.001873 **
library(FSA)
extraSS(model.8, model.9, model.10,
com=model.7)
Model 1: Longnose ~ Acerage + Maxdepth + NO3 + DO2
Model 2: Longnose ~ Acerage + Maxdepth + NO3 + SO4
Model 3: Longnose ~ Acerage + Maxdepth + NO3 + Temp
Model A: Longnose ~ Acerage + Maxdepth + NO3
DfO RSSO DfA RSSA Df SS F Pr(>F)
1vA 63 107234.38 64 107903.97 -1 -669.59 0.3934 0.5328
2vA 63 107898.06 64 107903.97 -1 -5.91 0.0035 0.9533
3vA 63 104955.97 64 107903.97 -1 -2948.00 1.7695 0.1882
lrt(model.8, model.9, model.10,
com=model.7)
Model 1: Longnose ~ Acerage + Maxdepth + NO3 + DO2
Model 2: Longnose ~ Acerage + Maxdepth + NO3 + SO4
Model 3: Longnose ~ Acerage + Maxdepth + NO3 + Temp
Model A: Longnose ~ Acerage + Maxdepth + NO3
DfO logLikO DfA logLikA Df logLik Chisq Pr(>Chisq)
1vA 63 -346.83881 64 -347.05045 -1 0.21164 0.4233 0.5153
2vA 63 -347.04859 64 -347.05045 -1 0.00186 0.0037 0.9513
3vA 63 -346.10863 64 -347.05045 -1 0.94182 1.8836 0.1699
# # #
Power analysis
See the Handbook for information on this topic.