极大似然估计的渐近正态性

结论

  假设$x_1,\cdots ,x_n$是来自$f_{\theta }(x)$的独立同分布样本，${\hat{\theta }}_{MLE}$是参数$\theta$的极大似然估计，那么$${\hat{\theta }}_{MLE}\dot{\sim }N(\theta ,\frac{1}{nI(\theta )})\text{\hspace{1em}(1)}$$其中，$I(\theta )$为$Fisher$信息量。

证明

  首先来看单样本的情况，即有样本$x$来自$f_{\theta }(x)$，则其似然函数为$$l_x(\theta )=log(f_{\theta }(x))\text{\hspace{1em}(2)}$$对$\theta$求导有$${\dot{l}}_x(\theta )=\frac{\partial }{\partial \theta }log(f_{\theta }(x))=\frac{{\dot{f}}_{\theta }(x)}{f_{\theta }(x)}\text{\hspace{1em}(3)}$$${\dot{l}}_x(\theta )$被称作得分函数，它的期望为：$$E({\dot{l}}_x(\theta ))={\int }_{\chi }\frac{{\dot{f}}_{\theta }(x)}{f_{\theta }(x)}{f}_{\theta }(x)dx={\int }_{\chi }{\dot{f}}_{\theta }(x)dx={\int }_{\chi }\frac{\partial }{\partial x}{f}_{\theta }(x)dx=\frac{\partial }{\partial x}{\int }_{\chi }{f}_{\theta }(x)dx=\frac{\partial }{\partial x}1=0\text{\hspace{1em}(4)}$$
$I(\theta )$为$Fisher$信息量，被定义为得分函数${\dot{l}}_x(\theta )$的方差：
I ( θ ) = E { l ˙ x ( θ ) − E ( l ˙ x ( θ ) ) } 2 (5) I(\theta)=E\{\dot{l}_x(\theta)-E(\dot{l}_x(\theta))\}^2\tag{5} I(θ)=E{ l˙x​(θ)−E(l˙x​(θ))}2(5)
而由$E({\dot{l}}_x(\theta ))=0$可知：
I ( θ ) = E { l ˙ x ( θ ) } 2 = E { f ˙ θ ( x ) f θ ( x ) } 2 (6) I(\theta)=E\{\dot{l}_x(\theta)\}^2=E\{\frac{\dot{f}_{\theta}(x)}{ {f}_{\theta}(x)}\}^2\tag{6} I(θ)=E{ l˙x​(θ)}2=E{ fθ​(x)f˙​θ​(x)​}2(6)
因此可记${\dot{l}}_x(\theta )$为：$${\dot{l}}_x(\theta )\sim (0,I(\theta ))\text{\hspace{1em}(7)}$$
接下来考虑得分函数的二阶导数${\ddot{l}}_x(\theta )$，即对式子$(3)$等号两边同时对$\theta$求导：
$${\ddot{l}}_x(\theta )=\frac{\partial }{\partial \theta }(\frac{{\dot{f}}_{\theta }(x)}{f_{\theta }(x)})=\frac{{\ddot{f}}_{\theta }(x)}{f_{\theta }(x)}-(\frac{{\dot{f}}_{\theta }(x)}{f_{\theta }(x)}{)}^2\text{\hspace{1em}(8)}$$
因此得分函数的二阶导数${\ddot{l}}_x(\theta )$的期望为： E { l ¨ x ( θ ) } = 0 − E { f ˙ θ ( x ) f θ ( x ) } 2 = − I ( θ ) (9) E\{\ddot{l}_x(\theta)\}=0-E\{\frac{\dot{f}_{\theta}(x)}{ {f}_{\theta}(x)}\}^2=-I(\theta)\tag{9} E{ l¨x​(θ)}=0−E{ fθ​(x)f˙​θ​(x)​}2=−I(θ)(9)
同样可记${\ddot{l}}_x(\theta )$为：$$-{\ddot{l}}_x(\theta )\sim (I(\theta ),J(\theta ))\text{\hspace{1em}(10)}$$
其中，$J(\theta )$为${\ddot{l}}_x(\theta )$的方差，我们这里不进行考虑。

  接下来讨论$n$个样本的情况，即$x_1,\cdots ,x_n$是来自$f_{\theta }(x)$的独立同分布样本，那么此时的联合密度函数为：$f_{\theta }(X)=\prod _{i=1}^n{f}_{\theta }(x_i)$，同样的，总的得分函数为：$${\dot{l}}_X(\theta )=\sum _{i=1}^n{\dot{l}}_{x_i}(\theta )\text{\hspace{1em}(11)}$$
根据$(7)$，每个${\dot{l}}_{x_i}(\theta )\sim (0,I(\theta ))$，结合样本之间是独立的，可知：$${\dot{l}}_X(\theta )\sim (0,nI(\theta ))\text{\hspace{1em}(12)}$$
类似的，有：$$-{\ddot{l}}_X(\theta )=\sum _{i=1}^n(-{\ddot{l}}_{x_i}(\theta ))\text{\hspace{1em}(13)}$$
同样的，根据$(10)$，每个$-{\ddot{l}}_{x_i}(\theta )\sim (I(\theta ),J(\theta ))$，因此有：$$-{\ddot{l}}_X(\theta )\sim (nI(\theta ),nJ(\theta ))\text{\hspace{1em}(14)}$$
根据定义，基于样本$x_1,\cdots ,x_n$，参数$\theta$的极大似然估计${\hat{\theta }}_{MLE}$满足最大化条件${\dot{l}}_X(\hat{\theta })=0$，对其在$\theta$处一阶泰勒展开有：$$0={\dot{l}}_X(\hat{\theta })\approx {\dot{l}}_X(\theta )+{\ddot{l}}_X(\theta )(\hat{\theta }-\theta )\text{\hspace{1em}(15)}$$
对其变形，有：$$\hat{\theta }\approx \theta -\frac{{\dot{l}}_X(\theta )}{{\ddot{l}}_X(\theta )}=\theta +\frac{\frac{{\dot{l}}_X(\theta )}{n}}{-\frac{{\ddot{l}}_X(\theta )}{n}}\text{\hspace{1em}(16)}$$
式$(12)$和中心极限定理表明：$$\frac{{\dot{l}}_X(\theta )}{n}\dot{\sim }N(0,\frac{I(\theta )}{n})\text{\hspace{1em}(17)}$$
式$(14)$和大数定律表明：$$-\frac{{\ddot{l}}_X(\theta )}{n}趋于常量I(\theta )\text{\hspace{1em}(18)}$$
综合式$(16),(17),(18)$，即可得到$$\hat{\theta }\dot{\sim }N(\theta ,\frac{1}{nI(\theta )})\text{\hspace{1em}(19)}$$此即$(1)$式，证毕。