PRML Notes 1 - Probability Theory in Polynomial Regression in the Introduction

introduction

  In the introduction, it is first pointed out that "the problem of finding patterns in data is a basic problem". The author gave an example through handwritten digit recognition: handwritten characters are varied, how can we correctly recognize handwritten digits?
  Some people may say that we can directly write the program to set rules for it. However, since there are always handwritten characters that we cannot consider, whenever we find new characters that cannot be recognized by the program, we must add new rules, which leads to rule proliferation. For example, the following two 2, it is difficult for us to manually set the rules.

  Or understand it this way, we hope to have a mapping that can map the input image into the corresponding number. The mapping function set by manual methods may be very large, so we need to explore a less labor-saving and more efficient method, that is, machine learning.

  In the machine learning method, there are N digital images { x 1 , ⋯ , x N } \{x_1,\cdots,x_N\}{ x1​,⋯,xN​} is called the training set, which is used to adjust the parameters of the mapping. The number corresponding to each image is known, using the target vector (target vector)$t$ to represent the category label of the number. The machine learning method can also be regarded as a mapping function$f\left(x\right)$ , he takes the image$x$ as input, vector$y$ is the output. The yyoutput here$y$ vector form and target vector$t$ has the same form.

1. Training phase: determine the function $The\; exact\; form\; of\; f\left(x\right)$ , trained with digital images and known labels.
2. Test phase: use the function $f\left(x\right)$ predicts labels for new images (test set) images.
3. Generalization problem: predictive performance on data outside the training set.
     Additionally the original digital image may need to be transformed into a new variable space (preprocessing). Such as converting an image to a vector. If the training images are preprocessed, the test images also need to be preprocessed in the same way.
   Applications in which training data samples contain input vectors and corresponding target vectors are called supervised learning problems. The training data consists of a set of input vectors $x$ without any corresponding target value. This is called an unsupervised problem. The problem of feedback learning technology is to find the appropriate action under the given conditions to maximize the reward.
     The introduction mainly introduces the three most important tools in this book: probability theory, decision theory, and information theory.

Polynomial Curve Fitting

  from $sin\left(2\pi \right)$ This function is uniformly sampled between 0 and 1. The sampling process adds noise that conforms to the Gaussian distribution. The training data set consists ofxxNNof$x$Composed of$N$ observations ,$x\equiv (x_1,\cdots ,x_N{)}^T$ , observed value$t$记作$t\equiv (t_1,\cdots ,t_N)$ .
  Now if we only have the training data set$x$ and the set of observations$t$ , how to predict new$\hat{x}$ target variable$\hat{t}$ . Due to the interference of data sampling, for a given$\hat{x}$，$\hat{t}$ Uncertain. The authors use polynomial functions for curve fitting. $$y(x,w)=w_0+w_{1}x+w_2{x}^2+\cdots +w_M{x}^M=\sum _{j=0}^{M}where$$ MM_$M$ is the order of the polynomial,$x^j$ meansxxjjof$x$$j$ power. Polynomial coefficient$w_0,\cdots ,w_M$The whole is recorded as a vector $w$ . Note: Although the polynomial function$y(x,w)$is$A\; nonlinear\; function\; of\; x$ , but it is the coefficientw \boldsymbol{w}A linear function of$w\; .$
  To find a suitable$The\; value\; of\; w$ needs to be realized by minimizing the error function. For example each data point$x_n$The predicted value $y(x_n,w)$and target value$t_n$The sum of squares of the difference. Minimize the error function to find the appropriate $\mathbf{w}$。 E ( w ) = 1 2 ∑ n = 1 N { y ( x n , w ) − t n } 2 E(\boldsymbol{w})=\frac{1}{2}\sum_{n=1}^N\{y(x_n,\boldsymbol{w})-t_n\}^2 E ( w )=21​n=1∑N​{ y(xn​,w)−tn​}2 This is a non-negative quantity, if all values ​​are predicted correctly, the error function$E\left(x\right)=0$ .
The error function is$The\; quadratic\; function\; of\; w$ , derivation can get a unique solution, this solution is the minimum value solution of the error function${\mathbf{w}}^{\ast }$ .
  There is another problem to be solved below, that is the polynomial order$The\; value\; of\; M$ , that is,w \boldsymbol{w}The length of the$w\; vector.$

1. $M=0$ horizontal line
2. $M=1$ straight line
3. $M=3$ The fitting results are relatively close to
4. $M=9$ The fitted curve oscillates violently, completely passing through all points, the error function is 0, and overfitting.
     It stands to reason that it should be$The\; bigger\; the\; M$ , the better. Why does it oscillate? Intuitively, the authors say, a larger$The\; polynomial\; of\; the\; M$ value is over-tuned, so that the polynomial is adjusted to match the random noise of the target value. Personally, I feel that the random noise should also be fitted during the adjustment, so the vibration is relatively large. However, as the training data set increases,MMThe unsatisfactory aspect of this result is that$we$ have to limit the number of parameters according to the size of the training set available. A more reasonable explanation is to choose the complexity of the model according to the complexity of the problem to be solved. Therefore, a method will be mentioned later, using the Bayesian model, the effective number of parameters will be automatically adjusted according to the size of the data set.

probability theory

  Sum rule: $$p(X)=\sum _{Y}p(X,Y)$$
  product rule:$$p(X,Y)=p(Y|X\left)p\right(X)$$
according to the product rule, and the symmetry$p(X,Y)=p(Y,X)$ , the relationship between two conditional probabilities can be obtained:
$$\begin{gathered} p(X,Y)=p(Y,X) \\ p(Y|X)p(X)=p(X|Y)p(Y) \\ p(Y|X)=\frac{p(X|Y)p(Y)}{p(X)} \end{gathered}$$
This is Bayes' theorem. According to the sum rule, the above result can be written as:
$$p(Y|X)=\frac{p(X|Y\left)p\right(Y)}{{\sum }_{Y}p(X,Y)}$$
The parameter w \boldsymbol{w}   in the polynomial curve fitting example$w$ , before we observe the data, for$w$ has some assumptions, which with prior probability$given\; in\; the\; form\; of\; p\left(w\right)$ . Observation dataD = { t 1 , ⋯ , t N } D=\{t_1,\cdots,t_N\}D={ t1​,⋯,tN​} The effect of the conditional probability$p(D|w)$ expression, using Bayes' theorem to represent the polynomial curve fitting problem:
$$p(\mathbf{w}|D)=\frac{p(D|\mathbf{w})p(\mathbf{w})}{p(D)}$$It allows us to pass the posterior probability $p(w|D)$ , after observingDDEstimatew \boldsymbol{w}$after\; D$Uncertainty of$w\; .$The p on the right side of the formula$p(D|w)$ (called the likelihood function) is a very interesting quantity, which represents the given parameterw \boldsymbol{w}The possibility of the appearance of$w$$The\; integral\; of\; w$ is not necessarily equal to one. All quantities in the formula are aboutw \boldsymbol{w}The function of$w$$p(D)=\int p(D|w\left)p\right(w)dw$ . Frequentist and Bayesian pairs with$There\; are\; different\; views\; on\; the\; value\; of\; w$ , the frequentist thinks that$w$ is a fixed parameter, which is commonly usedfor maximum likelihood estimation, while Bayesians believe that$w$ is expressed by a probability distribution, such asa bootstrap.
  The following considers the case of maximum likelihood estimation Gaussian distribution, for the unary real-valued variable$x$ ,empty infinitesimal:
N ( x ∣ µ , σ 2 ) = 1 ( 2 π σ 2 ) 1 2 exp { − 1 2 σ 2 ( x − µ ) 2 } \mathcal{N}(x|\ in,\sigma^2)=\frac{1}{(2\pi\sigma^2)^\frac{1}{2}}exp\{-\frac{1}{2\sigma^2}( x-\mu)^2\}N(x∣μ,p2)=( 2 p . p2)21​1​exp{ −2 p21​(x−m )2 }
Intuitive understanding is that given the mean value$\mu$ and variance$p^2$，$distribution\; of\; x$ . Here the mean and variance are also unknown.
  Now suppose we have an observation data set$\mathbf{t}=(t_1,\cdots ,t_N{)}^T$ , the observations of this data set are independently taken from a data set that conforms to a Gaussian distribution, so given the mean and variance, we can get the observation data set t \boldsymbol{t}The joint probability of each outcome in$t$
$$p(\mathbf{t}|\mu ,p^2)=\prod _{n=1}^N\mathcal{N}(t_n|\mu ,p^2)$$
Through the maximum likelihood estimation method, we can get the maximum likelihood solution of the mean:
$$m_{ML}=\frac{1}{N}\sum _{n=1}^N{t}_n$$
and the maximum likelihood solution of variance:
$$p_{ML}^2=\frac{1}{N}\sum _{n=1}N(t_n-m_{ML}{)}^{2Let}$$ E [ σ ML 2 ] = ( N − 1 N ) σ 2 \mathbb{E}[ \$E[p_{ML}^2]=(\frac{N-1}{N})p^2$ , so maximum likelihood estimation has the disadvantage of underestimating the variance (less problematic for larger N). In practical problems, since we are interested in models with many parameters, the maximum likelihood offset problem will be very serious. The maximum likelihood offset problem is at the heart of the overfitting problem encountered in the polynomial curve fitting problem above.
  Now back to the problem of curve fitting, there is$N$ inputs$\mathbf{x}=(x_1,\cdots ,x_N{)}^T$ and their corresponding target values$\mathbf{t}=(t_1,\cdots ,t_N{)}^T$ , assuming a given$The\; value\; of\; x$ , corresponding to$The\; t$ value obeys a Gaussian distribution, and the mean of the distribution is$y(x,w)$, so:
$$p(t|x,\mathbf{w},b)=\mathcal{N}(t|y(x,\mathbf{w}),b^{-1})$$, where$\beta$ is the precision, it is the reciprocal of the distribution variance,$b^{-1}$ is the variance. Determine the unknown parameter w \boldsymbol{w}by the maximum likelihood method$w$ sum$The\; value\; of\; \beta$，似然投行载：
$$p(\mathbf{t}|\mathbf{x},\mathbf{w},b)=\prod _{n=1}^N\mathcal{N}(t_n|y(x_n,\mathbf{w}),b^{-1})=\prod _{n=1}^N\frac{1}{(2p{)}^{\frac{1}{2}}{b}^{-\frac{1}{2}}}exp\frac{(t_n-y(x_n,\mathbf{w}){)}^2}{-2b^{-1}}$$, again emphasizing $b^{-1Specify}$ the equation:
$$\begin{gathered} \ln p(\mathbf{t}|\mathbf{x},\mathbf{w},b)=-\sum _{n=1}^N\frac{(t_n-y(x_n,\mathbf{w}){)}^2}{2b^{-1}}-\sum _{n=1}^N\ln (2\pi {)}^{\frac{1}{2}}-\sum _{n=1}^N\ln b^{-\frac{1}{2}} \\ =-\frac{}{2}\sum _{n=1}^N(t_n-y(x_n,w){)}^2-\frac{N}{2}\ln (2\pi )+\frac{N}{2}ln\beta \end{gathered}$$
由于 { y ( x n , w ) − t n } 2 = { t n − y ( x n , w ) } 2 \{y(x_n,\boldsymbol{w})-t_n\}^2=\{t_n-y(x_n,\boldsymbol{w})\}^2 { y(xn​,w)−tn​}2={ tn​−y(xn​,w)}2 , so the log-likelihood function is:
ln ⁡ p ( t ∣ x , w , β ) = − β 2 ∑ n = 1 N { y ( xn , w ) − tn } 2 + N 2 ln ⁡ β − N 2 ln ⁡ ( 2 π ) \ln{p}(\boldsymbol{t}|\boldsymbol{x},\boldsymbol{w},\beta)=-\frac{\beta}{2}\sum_{n= 1}^N\{y(x_n,\boldsymbol{w})-t_n\}^2+\frac{N}{2}\ln{\beta}-\frac{N}{2}\ln{( 2\pi)}lnp(t∣x,w,b )=−2bn=1∑N​{ y(xn​,w)−tn​}2+2N​lnb−2N​ln( 2 π ) Find the maximum likelihood solution${\mathbf{w}}_{ML}$, need to $w$ is derived, and the latter two terms are related to$w$ is irrelevant, only the pair− β 2 ∑ n = 1 N { y ( xn , w ) − tn } 2 -\frac{\beta}{2}\sum_{n=1}^N\{y(x_n ,\boldsymbol{w})-t_n\}^2−2b∑n=1N​{ y(xn​,w)−tn​}2 Find the maximum value,$\beta$ only plays a scaling role, so it is equivalent to$\frac{1}{2}{\sum }_{n=1}^N\left\{y\right(x_n,w)-t_n{\}}^2$ Find the minimum value. Interestingly, this returns to the original minimized error function. Under the assumption of Gaussian noise, the square sum error function is a natural result of maximizing the likelihood function. At this time, we have obtained the parameter$w$ .
  Using the maximum likelihood method can also get the variance$b_{ML}^{-1}$The maximum likelihood solution
β ML − 1 = 1 N ∑ n = 1 N { y ( xn , w ML ) − tn } 2 \beta_{ML}^{-1}=\frac{1}{N}\sum_ {n=1}^{N}\{y(x_n,\boldsymbol{w}_{ML})-t_n\}^2bML−1​=N1​n=1∑N​{ y(xn​,wML​)−tn​}2 Same as Gaussian distribution, still need to determine$w_{ML}$Then determine $b_{ML}^{-1}$. Now available for new $x$ has been predicted, and the prediction result is given byttThe prediction distribution of the probability distribution of$t$$$p(t|x,{\mathbf{w}}_{ML},b_{ML})=\mathcal{N}(t|y(x,w_{ML}),b_{ML}^{-1})$$。