Exponential distribution family and generalized linear models

1. Exponential distribution family

1.1 Definition

Exponential family distribution (The exponential family distribution), which is different from exponential distribution (exponential distribution). The exponential distribution family does not specifically refer to a distribution, but a general term for a series of distributions that meet the characteristics. In probability statistics, if a probability distribution satisfies the following formula, we call it an exponential family distribution.
$p(y;\theta)=b(y)exp\left(\eta(\theta)T (y)-A(\theta)\right)$
among them， $\eta$ is the natural parameter of the distribution (nature parameter); $T (y)$ is a sufficient statistic, usually $T (y) = y$ 。 $a(\eta)$ is the logarithmic partition function, $e^{-a(\eta)}$ plays a normalizing role in the formula to ensure that the probability density function is in the random variableThe integral over $y$ $If T, a, b$ are determined, a distribution can be determined, $\eta$ is a parameter.

Commonly used distributions such as normal distribution, Bernoulli distribution, exponential distribution, Poisson distribution, and gamma distribution all belong to the exponential distribution family.

1.2 Bernoulli distribution

We have the following equations:
$\begin{align}p(y;\theta)&=\theta^y(1-\theta)^{1-y}\\[ 2ex ] &=exp\left(y\log\theta+\left(1-y\right)\log\left(1-\theta\right)\right) \\[2ex] &=exp\left(\log \frac{\theta}{1-\theta}y+\log(1-\theta)\right)\end{align}$
We have the following equation:
$\begin{align} &b(y)=1 \\[2ex] &\eta(\theta)=\log\frac{\theta}{1-\theta}\\[2ex] &T(y)=y \\[2ex] &A(\theta)=-log(1-\theta)=log(1+e^{\eta(\theta)}) \end{align}$

1.3 Gaussian distribution

For the mean value $\mu$ with variance $\sigma$ Determine the equation for σ as a function of σ: $p$
$\begin{align} p(y;\mu,\sigma)&=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{( y-\mu)^2}{2\sigma^2}} \\[2ex] &=\frac{1}{\sqrt{2\pi}}e^{\eta(\mu,\sigma)T (y)-\log\sigma-\frac{\mu^2}{2\sigma^2}}\\[2ex]\end{align}$
Let us have the following equations for the inverse equation:
$\begin{align} &b(y)=\frac{1}{\sqrt{2\pi}} \\[2ex] &\eta(\sigma)=[\frac {\mu}{\sigma^2},-\frac{1}{2\sigma^2}]\\[2ex] &T(y)=[y,y^2] \\[2ex] &A(\ theta)=\frac{\mu^2}{2\sigma^2}+\log\sigma\end{align}$

1.4 Other exponential distribution families

Multinomial distribution (multinomial), used to model multivariate classification problems;
Poisson distribution (Poisson), used to model counting processes, such as the number of visitors to a website, the number of customers in a store, etc.;
Gamma distribution (gamma) and exponential distribution (exponential), used to model time intervals, such as waiting time, etc.;
Beta distribution (beta) and Dirichlet distribution (Dirichlet), for probability distribution;
Wishart distribution (Wishart), for covariance matrix distributions.

2. Generalized linear model (GLM)

The well-known linear regression and logistic regression belong to glm, in which linear regression assumes a Gaussian distribution, and logistic regression assumes a Bernoulli distribution, but it is not very clear why this is so.

2.1 Three assumptions

In the given argument $x$ and parameters $\theta$ , the dependent variable $y$ obeys the exponential distribution family
Given $x$ , the ultimate goal is to find $T (y)$ expectations $E [T (y) ∣ x]$
Natural parameter $\eta$ can be expressed as the independent variableThe linear relationship of $x$ $η=\theta^Tx$

A generalized linear model is fitted by fitting The conditional mean/expectation of $y$ $x$ and parameters $\theta$ given), and assuming $y$ fits a distribution in the family of exponential distributions, extending the standard linear model

2.2 Bernoulli distribution

For the Bernoulli distribution, because it is a binary classification problem, we choose $p(y|x;\theta) \sim Bernoulli(\Phi)$ is $\phi$ is the only parameter under the exponential distribution family. According to the above derivation:
$\begin{align} h_\theta(x) &= E[y|x;\theta] \\[2ex] & =\Phi \end{align}$

$\begin{align} \eta&=\log\frac{\phi}{1-\phi}\\[2ex] &=\theta^Tx \\[2ex] \end{align}$
Equation:
$\begin{align} y&=\frac{1}{1+e^{-\eta}}\\[2ex] &= \frac{1}{1+e^{-\theta^Tx}}\\[2ex]\end{align}$
The above formula is the expression of logistic regression, which corresponds to the assumption of Bernoulli distribution of y under logistic regression.

2.3 Gaussian distribution

For a Gaussian distribution, $The mean of y$ is the parameter $\mu$ , according to the above derivation:
$y=\mu=\eta=\theta^Tx (Assume \sigma=1)$
The above formula and linear regression forEchoes the assumption that $y is a Gaussian distribution$

3. GLM modeling process

According to the problem choose a distribution in the exponential distribution family as the pair assumption of $y$
$\eta$ under this distribution $η$ ， actually $\eta=\eta(w^T)$ , of which $w^T$ is the true parameter of the distribution, and $\eta$ is simply $w^T$ as a parameter
Calculate the expectation of this distribution, and use $\eta$ $y=\phi=\frac{1}{1+e^{−η}}$ in the above Bernoulli distribution $y = ϕ = \frac{1}{1 + e ^{- h}}$
$\eta=\theta^Tx$ according to the assumption of GLM $the = i^{T} x$ is the GLM model

4. Summary

Let us define the equation: $p(y;\theta)=b(y)exp\left(\ eta(\theta)T(y)-A(\theta)\right)$
Commonly used distributions such as normal distribution, Bernoulli distribution, exponential distribution, Poisson distribution, and gamma distribution all belong to the exponential distribution family.
A generalized linear model is fitted by fitting The conditional mean/expectation of $y$ $x$ and parameters $\theta$ given), and assuming $y$ fits a distribution in the family of exponential distributions, extending the standard linear model.

This article is only used as a personal learning record, not for commercial use, thank you for your understanding and cooperation.

Reference: https://shangzhih.github.io/zhi-shu-fen-bu-zu-he-yan-yi-xian-xing-hui-gui.html