Bayesian decision maximum likelihood estimation

Bayesian decision

        First look at Bayes classifier, we all know the classic Bayesian formula:

 

        Wherein: p (w): priori probability represents the probability distribution of each category; : class conditional probability represents the probability of some sort in the premise, something happens; and for the posterior probability, that an event has occurred and the probability that it belongs to a category, with the posterior probability, we can classify the sample. The larger posterior probability indicates a greater likelihood of something fall into this category, we have more reason to return to it under this category.

        We look at an intuitive example: Given: In summer, a park probability is 1/2 men wear sandals, women sandals probability is 2/3, and the park is usually male to female ratio is 2: 1, question: If you have a random encounter people wear sandals in the park, what is the probability of his gender as male or female respectively, how much?

        From the problem of view, is the talk of the above, something happened, it belongs to a category of probability is how much? That is the posterior probability.

        Assume:

        It can be obtained from known:

 

        Men and women wear sandals independent of each other, so

(If only consider the classification problem, we need only compare the size of the posterior probability, the value is not important).

        Bayesian calculated by the formula:

 

 

 

Leads to problems

        But in practical problems are not so lucky, we can get the data may be only a limited number of sample data, and the prior probability and conditional probability class (all types of population distribution) are unknown. When classified according to the only sample data, a viable option is that we need to first class prior probability and the conditional probability estimate, and then apply the Bayesian classifier.

        Estimated prior probability is relatively simple, 1, belongs to a natural state of each sample are known (supervised learning); 2, rely on experience; 3, with an estimated frequency of various types of training samples occur.

        The estimated conditional probability of class (very difficult) because comprising: a probability density function contains all the information of a random variable; sample data may be low; dimension feature vector x and the like can be large. In short, to directly estimate the conditional probability density function of the class is difficult. The solution is, completely unknown to the estimated probability density conversion of the estimated parameters. Here will be the probability density estimation problem into a problem of parameter estimation, maximum likelihood estimation is a kind of parameter estimation method. Of course, selecting the probability density function is very important, the model is correct, in the infinite sample area, we will get a more accurate estimate, if the models are wrong, it is estimated that half of the argument, certainly lacks meaning.

 

An important prerequisite

        Speaking of the above, the parameter estimation method is only a simplified process for solving practical problems (since the class conditional probability density function estimated directly is difficult). It is possible to use the maximum likelihood estimation method of sample must need to meet certain assumptions.

        An important prerequisite: the distribution of training samples representative of the true distribution of the sample. Each sample set of samples are so-called independent and identically distributed random variables (iid conditions), and there is adequate training samples.

 

Maximum likelihood estimate

        However, the principle of maximum likelihood estimation, with a picture to illustrate, as shown below:

 

        To sum up, the purpose of maximum likelihood estimation is: the use of a known sample results, thrust reversers are most likely (most probable) cause the value of this parameter results.

        Principle: maximum likelihood estimates are based on a statistical method of maximum likelihood, based on the principle of, it is the application of probability theory in statistics. Maximum likelihood estimation is provided a method to assess the observed data given the model parameters, namely: "model has been set, parameters are unknown." By several tests, the results observed, the results obtained using experimental value of a parameter enables the probability of occurrence of the maximum sample, it referred to as the maximum likelihood estimation.

        Since the sample set are independent and identically distributed samples, it may only be considered a Class D sample sets, to estimate the parameter vector θ. Sample set is known in mind:

 

        The likelihood function (linkehood function): joint probability density function is called with respect to the likelihood function of θ.

 

        如果是参数空间中能使似然函数最大的θ值,则应该是“最可能”的参数值,那么就是θ的极大似然估计量。它是样本集的函数,记作:

 

求解极大似然函数

        ML估计:求使得出现该组样本的概率最大的θ值。

 

         实际中为了便于分析,定义了对数似然函数:

        1. 未知参数只有一个(θ为标量)

        在似然函数满足连续、可微的正则条件下,极大似然估计量是下面微分方程的解:

        2.未知参数有多个(θ为向量)

        则θ可表示为具有S个分量的未知向量:

 

         记梯度算子:

 

         若似然函数满足连续可导的条件,则最大似然估计量就是如下方程的解。

 

         方程的解只是一个估计值,只有在样本数趋于无限多的时候,它才会接近于真实值。

 

极大似然估计的例子

        例1:设样本服从正态分布,则似然函数为:

 

        它的对数:

 

        求导,得方程组:

 

 

        联合解得:

 

        似然方程有唯一解:,而且它一定是最大值点,这是因为当时,非负函数。于是U和的极大似然估计为

 

        例2:设样本服从均匀分布[a, b]。则X的概率密度函数:

 

        对样本

 

        很显然,L(a,b)作为a和b的二元函数是不连续的,这时不能用导数来求解。而必须从极大似然估计的定义出发,求L(a,b)的最大值,为使L(a,b)达到最大,b-a应该尽可能地小,但b又不能小于,否则,L(a,b)=0。类似地a不能大过,因此,a和b的极大似然估计:

 

 

 

总结

        求最大似然估计量的一般步骤:

        (1)写出似然函数;

        (2)对似然函数取对数,并整理;

        (3)求导数;

        (4)解似然方程。

        最大似然估计的特点:

        1.比其他估计方法更加简单;

        2.收敛性:无偏或者渐近无偏,当样本数目增加时,收敛性质会更好;

        3.如果假设的类条件概率模型正确,则通常能获得较好的结果。但如果假设模型出现偏差,将导致非常差的估计结果。

 

转自:https://blog.csdn.net/zengxiantao1994/article/details/72787849

Guess you like

Origin www.cnblogs.com/virgil626903642/p/11505573.html