[Probability Theory] Final Review Notes: Univariate Linear Regression

1. Regression analysis

Relationships between variables may be deterministic (functional relationships) or statistical dependencies (correlations). In the correlation relationship, the "dependent variable" is a random variable, and its value has uncertainty. It cannot be analyzed by examining the functional relationship, but by statistical methods. There are two ways to examine the correlation relationship: when the independent variable is a non-random variable that can be measured and controlled, regression analysis is used ; if the independent variable is also a random variable or an uncontrollable variable, correlation analysis is used .

2. One-element linear regression model

Regression function : set $x$ is a controllable variable,$Y$ is the random variable associated with it. When the argumentxxWhen$x$$Y$ has a definite(conditional) distributioncorresponding to it. If$The\; mathematical\; expectation\; of\; Y$ exists, then its value varies withxxdepends on the value of$x$$The\; function\; of\; x$ , denoted as$\mu (x)$，即$m\left(x\right)=E(Y|x)$ , abbreviated as$\mu (x)$为$Y$ about$The\; regression\; function\; of\; x$ . $$\begin{gathered} \text{Functional relationship:}x\text{determined}⟶Y\text{'svalueuniquely determined} \\ \text{Regression analysis:}x\text{determined}⟶Y\text{'sdistributionThe only definite} \end{gathered}$$ basic task of regression analysis is to use experimental data to estimate$Y$ aboutxxRegression function of$x$$m\left(x\right)$ .

Univariate linear regression problem : Let $Y$ aboutxxThe regression function of$x$$\mu \left(x\right)$，若$\mu \left(x\right)$ is a linear function$m\left(x\right)=a+bx$ , then estimate$The\; problem\; of\; \mu \left(x\right)$ is called a linear regression problem of one variable.

Univariate linear regression model : Let $x$ is the controllable variable,$Y$ is dependent onxxInstead of$x$$$\{\begin{array}{l}Y=a+bx+e\\ e\text{~}N(0,p^2)\end{array}$$Among them, the unknown parameters $a,b,p^2$ does not depend on$x$ , the model is called a linear regression model.

样本：$(x_1,Y_1),(x_2,Y_2),\cdots ,(x_n,Y_n)$（$Y_1,Y_2,\cdots ,Y_n$are independent random variables)
sample values : $(x_1,y_1),(x_2,y_2),\cdots ,(x_n,y_n)$

A sample form of a linear regression model : $$\{\begin{array}{l}{Y}_i=a+b{x}_i+e_i\\ e_i\text{~}N(0,p^2)\end{array}(i=1,2,\cdots ,n),\text{且}{e}_1,e_2,\cdots ,e_n\text{Mutually independent}$$ empirical regression linear equation: if by$(x_1,y_1),(x_2,y_2),\cdots ,(x_n,y_n)$ get the unknown parameters$a,$Estimated value of$b$$\hat{a},\hat{b}$ , then for a given$x$ , we can take$\hat{y}=\hat{a}+\hat{b}x$works$m\left(x\right)=a+The\; estimated\; value\; of\; bx$ , and the equation$\hat{y}=\hat{a}+\hat{b}x$called$Y$ aboutxxThe empirical regression equation for$x\; .$

3. $a,b$和$p^2$ estimates

Least squares method : known variables $x,Y$ 's$n$ pairs of test data$(x_1,y_1),(x_2,y_2),\cdots ,(x_n,y_n)$ , of which$x_i$Not all the same, make the sum of squared deviations $Q(a,b)=\sum _{i=1}^n{(y_i-a-b{x}_i)}^2$ , choose parameters$a,bUse$ Q ( a , b$Q(a,b)$ the smallest, this method is called the least squares method.

In order to find $Q(a,$The minimum value of$b$$)$$\frac{\partial Q}{\partial a},\frac{\partial Q}{\partial b}$Both are $0$，即$$\{\begin{array}{l}\frac{\partial Q}{\partial a}=-2\sum _{i=1}^n(y_i-a-b{x}_i)=0\\ \frac{\partial Q}{\partial b}=-2\sum _{i=1}^n(y_i-a-b{x}_i)x_i=0\end{array}$$得方程组$$\{\begin{array}{l}already+b\sum _{i=1}^n{x}_i=\sum _{i=1}^n{y}_i\\ a\sum _{i=1}^n{x}_i+b\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{x}_i{y}_i\end{array}$$上式称为正规方程组。其系数行列式为$$\left|\begin{array}{cc}n& \sum _{i=1}^n{x}_i\\ \sum _{i=1}^n{x}_i& \sum _{i=1}^n{x}_i^2\end{array}\right|=n\sum _{i=1}^n{x}_i^2-{\left(\sum _{i=1}^n{x}_i\right)}^2=n\sum _{i=1}^n{\left(x_i-\overline{x}\right)}^2$$ because$x_i$Not exactly the same, so the coefficient determinant is not $0$ , so the normal system of equations has a unique solution, we get$a,b$的估计值为$$\{\begin{array}{l}\hat{b}=\frac{S_{xy}}{S_{xx}}=\frac{\bar{x}\bar{y}-\overline{xy}}{\overline{x^2}-{\overline{x}}^2}\\ \hat{a}=\overline{y}-\hat{b}\overline{x}\end{array}$$其中$S_{xy}=\sum _{i=1}^n\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)$，$S_{xx}=\sum _{i=1}^n{\left(x_i-\overline{x}\right)}^2$。

Therefore, the empirical regression equation sought is $\hat{y}=\hat{a}+\hat{b}x$. If$\hat{a}=\overline{y}-\hat{b}\overline{x}$Substituting into the empirical regression equation, the empirical regression linear equation is $\hat{y}=\overline{y}-\hat{b}\overline{x}+\hat{b}x=\overline{y}+\hat{b}\left(x-\overline{x}\right)$ , which shows that the empirical regression line equation passes through the geometric center of the scatter plot$\left(\overline{x},\overline{y}\right)$。

Find $p^2$的估计。令${\hat{y}}_i=\hat{a}+\hat{b}{x}_i(i=1,2,\cdots ,n)$，称$y_i-{\hat{y}}_i$for $x_i$处的残差，称$Q_e=\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2=\sum _{i=1}^n{\left(y_i-\hat{a}-\hat{b}{x}_i\right)}^2$ is the residual sum of squares. In fact$Q_e=Q\left(\hat{a},\hat{b}\right)$is$Q(a,b)$ the minimum value. It can be proved that$p^{}$The unbiased estimator of${}^2$$\widehat{p^2}=\frac{Q_e}{n-2}$。

$Q_e$It can also be calculated in another way: $Q_e=S_{yy}-{\left(\hat{b}\right)}^2{S}_{xx}$，其中$S_{yy}=\sum _{i=1}^n{\left(y_i-\overline{y}\right)}^2$。

Equations: $$\boxed{\begin{array}{rl}\hat{y}& =\hat{a}+\hat{b}x\\ \hat{b}& =\frac{S_{xy}}{S_{xx}}\\ \hat{a}& =\overline{y}-\hat{b}\overline{x}\\ \widehat{p^2}& =\frac{Q_e}{n-2}=\frac{1}{n-2}\left[S_{yy}-{\left(\hat{b}\right)}^2{S}_{xx}\right]\end{array}}$$

4. A model that can be transformed into a linear regression

1. $Y=a{e}^{\beta x}\cdot eh\_\ln e\text{~}N(0,p^2)$

Let $a,b,p^2$ is withxxThe unknown parameter of$x\; is\; irrelevant.$

Determine the infinitesimal equation $$\begin{array}{rl}\ln Y& =\ln a+\beta x+\ln e\\ Y^{\prime }& =a+bx+e^{\prime }\end{array}$$For $Y^{\prime }=\ln Y,a=\ln a,b=b,e^{\prime }=\ln eh$ .

2. $Y=a+\beta h(x)+eh\_e\text{~}N(0,p^2)$

Let $a,b,p^2$ is with$x-$ independent unknown parameter,$h\left(x\right)$ isxxKnown function of$x\; .$

令$a=a,b=b,x^{\prime }=h\left(x\right)$ , then transformed into a linear regression model$$Y=a+b{x}^{\prime }+e$$