6.6 Applications to linear models

本文为《Linear algebra and its applications》的读书笔记

For easy application of the discussion to real problems that you may encounter later in your career, we choose notation that is commonly used in the statistical analysis of scientific and engineering data. Instead of $A\mathbf{x}=\mathbf{b}$, we write $X\mathbf{\beta }=\mathbf{y}$ and refer to $X$ as the design matrix (设计矩阵), $\mathbf{\beta }$ as the parameter vector (参数向量), and $\mathbf{y}$ as the observation vector (参数向量).

Least-Squares Lines 最小二乘直线

The simplest relation between two variables $x$ and $y$ is the linear equation $y={\beta }_0+{\beta }_{1}x$. Experimental data often produce points $(x_1,y_1),...,(x_n,y_n)$ that, when graphed, seem to lie close to a line. We want to determine the parameters ${\beta }_0$ and ${\beta }_1$ that make the line as “close” to the points as possible.

Suppose ${\beta }_0$ and ${\beta }_1$ are fixed, and consider the line $y={\beta }_0+{\beta }_{1}x$ in Figure 1. Corresponding to each data point $(x_j,y_j)$ there is a point $(x_j,{\beta }_0+{\beta }_1{x}_j)$ on the line with the same $x$-coordinate. We call $y_j$ the $observed$ value of $y$ and ${\beta }_0+{\beta }_1{x}_j$ the $predicted$ y-value. The difference between an observed $y$-value
and a predicted $y$-value is called a $residual(余差)$.

There are several ways to measure how “close” the line is to the data. The usual choice (primarily because the mathematical calculations are simple) is to add the squares of the residuals. The least-squares line is the line $y={\beta }_0+{\beta }_{1}x$ that minimizes the sum of the squares of the residuals. This line is also called a line of regression of $y$ on $x$ ($y$ 对 $x$ 的回归直线), because any errors in the data are assumed to be only in the $y$-coordinates. The coefficients ${\beta }_0,{\beta }_1$ of the line are called (linear) regression coefficients(回归系数).

If the measurement errors are in $x$ instead of $y$, simply interchange the coordinates of the data before plotting the points and computing the regression line. If both coordinates are subject to possible error, then you might choose the line that minimizes the sum of the squares of the orthogonal (perpendicular) distances from the points to the line.

If the data points were on the line, the parameters ${\beta }_0$ and ${\beta }_1$ would satisfy the equations

We can write this system as

This is a least-squares problem. The square of the distance between the vectors $X$ and $\mathbf{y}$ is precisely the sum of the squares of the residuals. Computing the least-squares solution of $X\mathbf{\beta }=\mathbf{y}$ is equivalent to finding the $\mathbf{\beta }$ that determines the least-squares line in Figure 1.

A common practice before computing a least-squares line is to compute the average $\overline{x}$ of the original $x$-values and form a new variable $x^{\ast }=x-\overline{x}$. The new $x$-data are said to be in mean-deviation form (平均偏差形式). In this case, the two columns of the design matrix will be orthogonal. Solution of the normal equations is simplified.

EXERCISES 14
Show that the least-squares line for the data $(x_1,y_1),...,(x_n,y_n)$ must pass through $(\overline{x},\overline{y})$. That is, show that $\overline{x}$ and $\overline{y}$ satisfy the linear equation $\overline{y}={\hat{\beta }}_0+{\hat{\beta }}_1\overline{x}$.
SOLUTION
Derive this equation from the vector equation $\mathbf{y}=X\hat{\mathbf{\beta }}+\mathbf{ϵ}$. Denote the first column of $X$ by $1$. Use the fact that the residual vector $\mathbf{ϵ}$ is orthogonal to the column space of $X$ and hence is orthogonal to $1$. Thus ${\sum }_{i=1}^{i=n}{ϵ}_i=0$.
$$\begin{array}{rl}\because y_i& ={\hat{\beta }}_0+x_i{\hat{\beta }}_1+{ϵ}_i\\ \therefore \sum _{i=1}^{i=n}{y}_i& =n{\hat{\beta }}_0+{\hat{\beta }}_1\sum _{i=1}^{i=n}{x}_i\\ \therefore \overline{y}& ={\hat{\beta }}_0+{\hat{\beta }}_1\overline{x}\end{array}$$

---

Given data for a least-squares problem, $(x_1,y_1),...,(x_n,y_n)$, the following abbreviations are helpful:
$$\begin{gathered} \sum x=\sum _{i=1}^n{x}_i,\sum x^2=\sum _{i=1}^n{x}_i^2, \\ \sum y=\sum _{i=1}^n{y}_i,\sum xy=\sum _{i=1}^n{x}_i{y}_i \end{gathered}$$

The normal equations for a least-squares line $y={\hat{\beta }}_0+{\hat{\beta }}_{1}x$ is $X^{T}X\mathbf{\beta }=X^T\mathbf{y}$.
$$\because X^{T}X=\left[\begin{array}{c}{1}^T\\ {\mathbf{x}}^T\end{array}\right]\left[\begin{array}{cc}1& \mathbf{x}\end{array}\right]=\left[\begin{array}{cc}n& \sum x\\ \sum x& \sum x^2\end{array}\right]$$The normal equation may be written in the form
$$\left[\begin{array}{cc}n& \sum x\\ \sum x& \sum x^2\end{array}\right]\hat{\mathbf{\beta }}=\left[\begin{array}{c}{1}^T\\ {\mathbf{x}}^T\end{array}\right]\mathbf{y}=\left[\begin{array}{c}\sum y\\ \sum xy\end{array}\right]$$$$\therefore n{\hat{\beta }}_0+{\hat{\beta }}_1\sum x=\sum y,{\hat{\beta }}_0\sum x+{\hat{\beta }}_1\sum x^2=\sum xy$$

If $X$ has 2 linearly independent columns, then
$$\begin{array}{rl}\hat{\mathbf{\beta }}& ={\left[\begin{array}{cc}n& \sum x\\ \sum x& \sum x^2\end{array}\right]}^{-1}\left[\begin{array}{c}\sum y\\ \sum xy\end{array}\right]\\ & =\frac{1}{n\sum x^2-(\sum x{)}^2}\left[\begin{array}{cc}\sum x^2& -\sum x\\ -\sum x& n\end{array}\right]\left[\begin{array}{c}\sum y\\ \sum xy\end{array}\right]\end{array}$$$$\therefore {\hat{\beta }}_0=\frac{\sum x^2\sum y-\sum x\sum xy}{n\sum x^2-(\sum x{)}^2},{\hat{\beta }}_1=\frac{n\sum xy-\sum x\sum y}{n\sum x^2-(\sum x{)}^2}$$

---

Consider the following numbers.

(i) ${\Vert X\hat{\mathbf{\beta }}\Vert }^2$—the sum of the squares of the “regression term.” Denote this number by $SS(R)$.
(ii) ${\Vert \mathbf{y}-X\hat{\mathbf{\beta }}\Vert }^2$—the sum of the squares for error term. Denote this number by $SS(E)$.
(iii) ${\Vert \mathbf{y}\Vert }^2$—the “total” sum of the squares of the $y$-values. Denote this number by $SS(T)$.

Every statistics text that discusses regression and the linear model $\mathbf{y}=X\mathbf{\beta }+ϵ$ introduces these numbers.

EXERCISES 19
Justify the equation $SS(T)=SS(R)+SS(E)$. This equation is extremely important in statistics, both in regression theory and in the analysis of variance.
SOLUTION
This follows from the Pythagorean Theorem (in Section 6.1).

Then $$\begin{array}{rl}SS(E)& =SS(T)-SS(R)\\ & ={\Vert \mathbf{y}\Vert }^2-{\Vert X\hat{\mathbf{\beta }}\Vert }^2\\ & ={\mathbf{y}}^T\mathbf{y}-{\hat{\mathbf{\beta }}}^T{X}^{T}X\hat{\mathbf{\beta }}\\ & ={\mathbf{y}}^T\mathbf{y}-({\hat{\mathbf{\beta }}}^T{X}^{T}X\hat{\mathbf{\beta }}+{\hat{\mathbf{\beta }}}^T{X}^T\mathbf{ϵ})\\ & ={\mathbf{y}}^T\mathbf{y}-{\hat{\mathbf{\beta }}}^T{X}^T(X\hat{\mathbf{\beta }}+\mathbf{ϵ})\\ & ={\mathbf{y}}^T\mathbf{y}-{\hat{\mathbf{\beta }}}^T{X}^T\mathbf{y}\end{array}$$

This is the standard formula for $SS(E)$.

---

The General Linear Model

In some applications, it is necessary to fit data points with something other than a straight line. In the examples that follow, the matrix equation is still $X\mathbf{\beta }=\mathbf{y}$, but the specific form of $X$ changes from one problem to the next. Statisticians usually introduce a residual vector(余差向量) $ϵ$, defined by $ϵ=\mathbf{y}-X\mathbf{\beta }$, and write
$$\mathbf{y}=X\mathbf{\beta }+ϵ$$

Any equation of this form is referred to as a linear model. Once $X$ and $\mathbf{y}$ are determined, the goal is to minimize the length of $ϵ$, which amounts to finding a least-squares solution of $X\mathbf{\beta }=\mathbf{y}$. In each case, the least-squares solution $\hat{\mathbf{\beta }}$ is a solution of the normal equations
$$X^{T}X\mathbf{\beta }=X^T\mathbf{y}$$

Least-Squares Fitting of Other Curves 其他曲线的最小二乘拟合

The next example show how to fit data by curves that have the general form
$$y={\beta }_0{f}_0(x)+{\beta }_1{f}_1(x)+...+{\beta }_k{f}_k(x)(2)$$

where $f_0,...,f_k$ are known functions and ${\beta }_0,...,{\beta }_k$ are parameters that must be determined.

As we will see, equation (2) describes a linear model because it is linear in the unknown parameters.

EXAMPLE 2
Suppose we wish to approximate the data by an equation of the form
$$y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2(3)$$Describe the linear model that produces a “least-squares fit” of the data by equation (3).
SOLUTION

The design matrix above is a Vandermonde matrix (范德蒙德矩阵)

Example 5 in Section 2.1 and Theorem 14 in Section 6.5 shows that if at least $3$ of the values $x_1,\dots ,x_n$ are distinct, then the least-squares solution $\hat{\mathbf{\beta }}$ will be unique.

Multiple Regression 多重回归

Suppose an experiment involves two independent variables(独立变量)—say, $u$ and $v$—and one dependent variable(相关变量), $y$. A simple equation for predicting $y$ from $u$ and $v$ has the form
$$y={\beta }_0+{\beta }_{1}u+{\beta }_{2}v(4)$$

A more general prediction equation might have the form
$$y={\beta }_0+{\beta }_{1}u+{\beta }_{2}v+{\beta }_3{u}^2+{\beta }_{4}uv+{\beta }_5{v}^2(5)$$

Equations (4) and (5) both lead to a linear model because they are linear in the unknown parameters (even though $u$ and $v$ are multiplied). In general, a linear model will arise whenever $y$ is to be predicted by an equation of the form
$$y_0={\beta }_0{f}_0(u,v)+{\beta }_1{f}_1(u,v)+...+{\beta }_k{f}_k(u,v)$$