[Machine Learning] Basic Probability Theory and Information Theory

For the original markdown link of this blog post , click here! !

introduction

Probability theory is a framework for representing statements of uncertainty. It not only provides methods to quantify uncertainty, but also provides axioms for deriving new uncertainty statements.

In the field of artificial intelligence, probability theory is used in two ways. First, the laws of probability tell us how AI systems reason, and based on this, some algorithms are designed to calculate or estimate expressions derived from probability theory. Second, probability and statistics can be used to analyze and evaluate the behavior of AI systems.

Probability theory allows us to make uncertain statements and to reason about the presence of uncertainty, while information theory allows us to quantify the amount of uncertainty in a probability distribution.

Machine learning often has to deal with uncertain quantities, and sometimes may also deal with random quantities. Uncertainty comes from three sources: the inherent randomness of the system being modeled, incomplete observations, and incomplete modeling.

probability theory

probability

• Bayesian genre probability : think that probability is the degree of certainty of something happening, that is, the degree of certainty;

• Frequency genre probability : Thinks that probability is just how often things happen.

• Random variable : A variable that randomly takes different values. Generally, plain lowercase letters are used to represent random variables, and italic lowercase letters are used to represent the values ​​that random variables can take. Such as random variable $The\; values\; that\; x$ can take are$x_1,x_2$。

  If the random variable is in vector form, use bold $x$ means that its possible value is$x$ . On its own, a random variable is simply a description of possible states, and it must specify the likelihood of each state through a probability distribution.

  Random variables may be discrete or continuous. Discrete random variables have finite or countably infinite number of states . States do not have to be integers, there may be some named states without values , and continuous random variables are real values .

Probability distributions

Discrete Random Variables and Probability Mass Functions

For discrete random variables, the probability distribution can use the probability mass function $(\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{m}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{f}\mathrm{u}\mathrm{n}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n},PMF)$ to describe, usually with capital letters$P$ means that the probability mass function maps each state that each random variable can take to the probability when the random variable takes that state. Such as random variable$x$ takes$The\; probability\; at\; time\; x$ is expressed as$P(\text{x}=x)$。

In general, a random variable x \rm xThe probability distribution P ( x ) P(\rm x)followed by$x$$P(\mathrm{x})$用：$\text{x}\sim P\left(\text{x}\right)$ to represent. Probability mass function$P\left(x\right)$ must satisfy the following conditions:

$⧫$ P P The domain of$P$ must be x \text{x}The collection of all states of$\text{x ;}$

$⧫$ $\forall x\in \text{x}，0\le P(\text{x})\le 1$;

$⧫$${\sum }_{x\in \text{x}}P\left(x\right)=1$.

Suppose a random variable $\text{x}$ has$k$ states, and follow a uniform distribution$\left(uniformdistribution\right)$ , by converting its$PMF\; \backslash$ rm$PMF$$PMF$ Set
$$P(\text{x}=x_i)=\frac{1}{k}$$
to all $i$ is established.

• joint probability distribution

  Apply the probability mass function to multiple random variables at the same time, such as $P(\text{x}=x,\text{y}=y)$，表示$\mathrm{x}=x,y=The\; probability\; that\; y$ occurs at the same time.

Continuous Random Variables and Probability Density Functions

For continuous random variables, the probability distribution can be used as probability density function $(\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{f}\mathrm{u}\mathrm{n}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n},PDF)\; description,\; usuallywith$ lowercase$letters$$p$ to indicate that it must satisfy:

$⧫$ P P The domain of$P$ must be x \text{x}The collection of all states of$\text{x ;}$

$⧫$ $\forall x\in \text{x}，0\le P(\text{x})$;

$⧫$${\int }_{-\infty }^{\infty }p(x)\mathrm{d}x=1$.

So you can use $PDFisintegrated$ to obtain the true probability mass of the point set.

For example, at $(a,b)$ ,
$$U(x;a,b)=\frac{1}{b-a}$$

marginal probability distribution

Definition : The probability distribution of a subset of the joint probability distribution of a set of variables. A probability distribution defined on a subset is called a marginal probability distribution .

Suppose there are discrete random variables $\text{x, y}$ and known$P(\text{x,y})$。那么
$$\forall x\in \text{x},P(\text{x}=x)=\sum _{y}P(\text{x}=x,\text{y}=y)$$
For continuous variables, use integral
$$p(x)=\int p(x,y)dy$$

conditional probability distribution

Definition : The probability of an event occurring given the occurrence of other events .

For example, given $\text{x}=x,$事件$\text{y}=The\; probability\; of\; y$ happening is
$$P(\text{and}=y\mid \text{x}=x)=\frac{P(\text{and}=y,\text{x}=x)}{P(\text{x}=x)}$$

The chain rule of conditional probability

The joint probability distribution of any multidimensional random variable can be decomposed into the form of multiplying the conditional probability of only one variable:
$$P({\text{x}}^{(1)},\cdots ,{\text{x}}^{(n)})=P({\text{x}}^{(1)})\prod _{i=2}^{n}P({\text{x}}^{(i)}\mid {\text{x}}^{\left(1\right)},\cdots ,{\text{x}}^{Let\; (i-1)})$$
determine,
$$P(a,b,c)=P(a)P(b\mid a)P(c|a,b)$$
The solution
$$\begin{array}{rl}P(a,b,c)& =P(a|b,c)P(b,c)\\ P(b,c)& =P(b|c)P(c)\\ P(a,b,c)& =P(a\mid b,c)P(b|c)P(c)\end{array}$$

Independence and Conditional Independence

Two random variables $\text{x}$和$\text{y}$ , if their joint probability distribution can be expressed as the product of individual probability distributions, then these two random variables are said to be independent of each other, that is,
$$\forall x\in \text{x},y\in \text{y},P(\text{x}=x,\text{y}=y)=P(\text{x}=x)P(\text{y}=y)$$
if the random variable$\text{x}$和$\text{The conditional probability distribution of y}$ for the random variable$\text{Each value of z}$ can be written in the form of a product, then the random variable$\text{x}$和$\text{y}$ in a given random variable$\text{z}$时是条件独立的，即
$$\forall x\in \text{x},y\in \text{y},z\in \text{z},P(\text{x}=x,\text{y}=y\mid \text{z}=z)=P(\text{x}=x|\text{z}=z)P(\text{y}=y|\text{z}=z)$$

Expectation, Variance, and Covariance

expect

期望$\left(Expectation\right)$ :$function$$f$ onthe probability distribution$P\left(\text{x}\right)$ or$The\; expectation\; of\; P\left(\text{x}\right)$ is expressed as,$x$ , then$f$ acts onxxAfter$x$$Mean\; value\; of\; f\left(x\right)$ . That is
$${\mathrm{E}}_{x\sim P}[f(x)]=\sum _{x}P\left(x\right)f\left(x\right)$$
is a continuous variable obtained by integral.

variance

Variance $\left(Variance\right)$ : when pair$When\; x$ is sampled according to its probability distribution, the random variablexxFunction valuef ( x ) f(x)$of\; x$$f\left(x\right)$ willshow how much difference. That is,
$$\mathrm{V}\mathrm{a}\mathrm{r}(f(x))=[(f(x)-\mathrm{E}[f(x)]{)}^2]$$
The square root is called the standard deviation.

Covariance

Covariance $\left(Covariance\right):\; Inasense$ , it givesthe linear correlation strength between variables and the scale of each variable. That is,
C ov ( f ( x ) , g ( y ) ) = E { ( f ( x ) − E [ f ( x ) ] ) ( g ( y ) − E [ g ( y ) ] ) } \mathrm{Cov }(f(x), g(y)) = \mathbb{E} \{ (f(x) - \mathbb{E}[f(x)]) (g(y) - \mathbb{E}[ g(y)]) \}C o v ( f ( x ) ,g(y))=E{ (f(x)−E[f(x)])(g(y)−E [ g ( y ) ] ) }
A large absolute value of the covariance means that the values ​​of the variables vary greatly and that they are far from their respective means. Ifthe covariance is positive, then both variables tend to take relatively large values ​​at the same time. Ifthe covariance is negative, then one of the variables tends to take relatively large values ​​while the other variable tends to take relatively small values, and vice versa.

Correlation coefficient $\left(correlation\right)$ normalizes the contribution of each variable, in order toonly measure the correlation of variables without being affected by the scale of each variable.

• covariance matrix

  Random vector $\mathbf{x}\in {\mathbb{R}}^{}$The covariance matrix of${}^n$$n\times n$ matrix, and satisfy
  $$Cov(x{)}_{i,j}=Cov({\text{x}}_i,{\text{x}}_j)$$
  where the diagonal elements are variances.

import numpy as np

x = np.array([1, 2, 3, 4, 5, 6, 7, 8])
y = np.array([8, 7, 6, 5, 4, 3, 2, 1])
Mean = np.mean(x)
Var = np.var(x)
Var_unbias = np.var(x, ddof=1)  # N - ddof
Cov = np.cov(x, y)
print(Mean)  # 4.5
print(Var)  # 5.25
print(Var_unbias)  # 6.0
print(Cov)
# [[ 6. -6.]
#  [-6.  6.]]

Common Probability Distributions

Bernoulli distribution

The distribution of a single binary random variable is also called the two-point distribution . $\varphi$ gives a random variable equal to$1$的probability。
$$\begin{gathered} P(\text{x}=1)=\varphi \\ P(\text{x}=0)=1-\varphi \\ P(\text{x}=x)={\varphi }^x(1-){\_}^{(1-x)} \\ {\mathbb{E}}_{\text{x}}[\text{x}]=\varphi \\ {Weare\_}_{\text{x}}(\text{x})=\varphi (1-)\_ \end{gathered}$$

Gaussian distribution

Also called normal distribution $\left(normaldistribution\right)N\; (x;\mu ,$ σ 2) = 1 2 π σ 2 e$-$$($$x$$-$$\mu$$)$$2$$2$$\sigma$$2$$\backslash$$mathcal$$\{$
$$\mathcal{N}(x;m,p^2)=\sqrt{\frac{1}{2p.s{\_}^2}}{e}^{-\frac{(x-\mu {)}^2}{2p^2}}$$
Sometimes $b=\frac{1}{p^2}$Indicates the precision of the distribution . The central limit theorem holds that the sum of a large number of independent random variables approximates a normal distribution, so the noise can be considered to be normally distributed .

index distribution

$$f(x)=\{\begin{array}{ll}\lambda e^{-\lambda x},& x>0\\ 0,& \text{other.}\end{array}$$

The default value
$$p(x;l)=l{1}_{x\ge 0}{e}^{-The\; \lambda x}$$
exponential distribution usesan indicator function$1_{x\ge 0}$to make when $When\; x$ takes a negative value, the probability is$0$。

$Laplace$ distribution

allow at any point $\mu is$ infinitesimal
$$p(x;c,m)=\frac{1}{2c}{e}^{-\frac{|x-\mu |}{c}}$$

Common functions

$sigmoid$ function

$$\sigma \left(x\right)=\frac{1}{1+e^{-x}}$$

This function is usually used to generate the parameter ϕ \phi in the Bernoulli distribution$\varphi$ , since its range is$(0,1)$ . Saturation occurs when a function's arguments become large or small, meaning it.

$softplus$ function

$$z\left(x\right)=\log (1+e^x)$$

该函数名来源于另一个函数$\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}$
x + = max ⁡ { 0 , x } x^+ = \max\{0, x\} x+=max{ 0,x}
该函数可用于产生正态分布的参数$\mu ,\sigma$，其范围是$(0,\infty )$。

贝叶斯规则

已知$P(\text{y}\mid \text{x})$计算$P(\text{x}\mid \text{y})$.
$$P(\text{x}\mid \text{y})=\frac{P(\text{and}|\text{x})P(\text{x})}{P\left(\text{and}\right)}=\frac{P(\text{and}|\text{x}\left)P\right(\text{x})}{{\sum }_{x}P(\text{and}|\text{x}=x)P(\text{x}=x)}$$
without knowing $The\; value\; of\; P\left(\text{y}\right)$ .

information theory

Thought: A less likely event contains more information than a more likely event.

The information needs to meet three conditions:

• Events that are very likely to occur should be less informative , and in extreme cases, events that are guaranteed to occur should be non-informative.
• Less likely events are more informative ;
• Independent events should have incremental information . For example, a coin toss that lands heads twice should convey twice as much information as a coin toss that lands heads once.

Self-information : an event $\text{x}=$The self-information of$x$
$$I(x)=-\log P\left(x\right)$$
with$\log$ to represent the natural logarithm, its base$\mathrm{is}$$e$ . The unit of information is nats$\left(nats\right)$ . If$2$ as the base, the unit is bits$(\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s})$。

Shannon Entropy : Used to quantify the total amount of uncertainty in an entire probability distribution. That is, the optimal code length in coding theory.
$$H(\text{x})=-{\mathbb{E}}_{x\sim P(x)}[\log P(x)]=-\sum _{x}P(x)\log P\left(x\right)$$

# 香农熵 plot
p = np.linspace(1e-6, 1 - 1e-6, 100)
entropy = - p * np.log(p)
plt.figure(figsize=(6, 4))
plt.plot(p, entropy)
plt.xlabel('p')
plt.ylabel('entropy(nats)')
plt.savefig('entropy.jpg', dpi=300)
plt.show()

def Shannon_Entropy(string):
    """
    计算最优编码长度
    :param string:
    :return:
    """
    entropy = 0
    for ch in range(0, 256):
        Px = string.count(chr(ch)) / len(string)
        if Px > 0:
            entropy += -Px * math.log(Px, 2)
    return entropy


message = "".join([chr(random.randint(0, 64)) for i in range(100)])
print(message)
# .1??@41-'7;<;!
# / +
# =;9!<"6#0%3
print(Shannon_Entropy(message))
# 5.419819909835648

Joint entropy : consider the entropy of multiple events at the same time, multiple random variables
$$H(X,Y)=-\sum _{x,y}P(x,y)\log P(x,y)$$
条件熵：一件事情发生的情况下，另一个事件的熵，
$$H(X\mid Y)=-\sum _{y}P(y)\sum _{x}P(x\mid y)\log P(x\mid y)$$
互信息：表示两件事情信息相交的部分。
$$I(X,Y)=H(X)+H(Y)-H(X,Y)$$
Information variation: Indicates the disjoint part of the information of two things.
$$V(X,Y)=H(X,Y)-I(X,Y)$$
$KL$ divergence: used to measure two distributions$P\left(\text{x}\right)$ andQ ( x ) Q(\text x)The gap between$Q$$($$\text{x}$$)$
$$D_{\mathrm{K}\mathrm{L}}(P\mid |Q)={\mathbb{E}}_{\text{x}\sim P}\left[\log \frac{P(x)}{Q(x)}\right]=\sum P(x)\log \frac{P\left(x\right)}{Q\left(x\right)}$$

$$D_{KL}(Q||P)=E_{\text{x}\sim Q}\left[\log \frac{Q(x)}{P\left(x\right)}\right]=\sum Q\left(x\right)\log \frac{Q\left(x\right)}{P\left(x\right)}$$

For example, $D_{KL}(P||Q)\ne D_{KL}(Q||P)$。

$★$ Note: Under discrete random variables,$The\; KL$ divergencemeasuredwhen we use a method designed to make the probability distribution$The\; minimum\; encoding\; of\; the\; length\; of\; the\; message\; generated\; by\; Q$ , sent containing the probability distributionPPThe amount of additional information required when$P\; generates\; a\; symbolic\; message.$

from scipy.stats import entropy  # 内置KL散度
import numpy as np


def KL_Divergence(p, q):
    """D(P || Q)"""
    return np.sum(np.where(p != 0, p * np.log(p / q), 0))


p = np.array([0.1, 0.9])
q = np.array([0.1, 0.9])

print(KL_Divergence(p, q))  # 0.0

print(entropy(p, q))  # 0.0  
# 两个分布一样

from scipy.stats import norm, entropy
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(1, 10, 1000)
y1 = norm.pdf(x, 3, 0.4)
y2 = norm.pdf(x, 6, 0.4)
P = y1 + y2  # 构造p(x)

KL_PQ = []
KL_QP = []
Q_list = []
for mu in np.linspace(0, 10, 100):
    for sigma in np.linspace(0.1, 5, 50):
        Q = norm.pdf(x, mu, sigma)
        Q_list.append(Q)
        KL_PQ.append(entropy(P, Q))
        KL_QP.append(entropy(Q, P))
# 寻找Q使得KL_PQ最小
KL_PQ_min = np.argmin(KL_PQ)
# 寻找Q使得KL_QP最小
KL_QP_min = np.argmin(KL_QP)

plt.rcParams['font.family'] = ["Times New Roman"]
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].plot(x, P / 2, 'g-', label="$\\bf P(x)$")
axes[0].plot(x, Q_list[KL_PQ_min], 'b-.', label="$\\bf {Q^{*}(x)}$")
axes[0].set_ylim(0, 1.2)
axes[0].set_xlabel("$\\bf x$")
axes[0].set_ylabel("$\\bf P(x)$")
axes[0].set_title("$\\bf \\rm \\arg\\min_{Q}(KL(P||Q))$")
axes[0].legend()

axes[1].plot(x, P / 2, 'g-', label="$\\bf P(x)$")
axes[1].plot(x, Q_list[KL_QP_min], 'r-.', label="$\\bf {Q^{*}(x)}$")
axes[1].set_ylim(0, 1.2)
axes[1].set_xlabel("$\\bf x$")
axes[1].set_ylabel("$\\bf P(x)$")
axes[1].set_title("$\\bf \\rm \\arg\\min_{Q}(KL(Q||P))$")
axes[1].legend()

plt.savefig("./images/KL.jpg", dpi=300)
plt.show()

Cross Entropy : A and $A\; closely\; related\; quantity\; to\; KL$$divergence$ is cross-entropy. The meaning is the actual output probability$Q$ and expected output probabilityPPthe distance between$P.$
$$\begin{array}{rl}H(P,Q)& =H(P)+D_{\mathrm{K}\mathrm{L}}(P\mid \mid Q)=-{\mathbb{E}}_{x\sim P}[\log Q(x)]\\ & =-\sum _{x}P(x)\log Q(x)\end{array}$$
For binary classification , the cross-entropy calculation method is as follows
$$H(P,Q)=-\frac{1}{N}\sum _{x}P\left(x\right)\log Q(x)+(1-P(x))\log (1-Q(x))$$

其中，$P(x)$表示样本$x$的期望(真实)标签，正类为1，负类为0；$Q(x)$表示模型实际预测该样本$x$类别为正的概率。$N$表示样本个数。

假设有$3$个样本，其真实类别标签是$P=(1,0,1)$ ，实际预测样本为正的概率为$Q=(0.8,0.1,0.7)$。计算期望输出和实际输出之间的交叉熵，如下

from sklearn.metrics import log_loss
import numpy as np

P = np.array([1, 0, 1])
Q1 = np.array([0.8, 0.1, 0.7])
Q2 = np.array([0.1, 0.7, 0.3])
# binary_ent = 0.0
# for i in range(P.size):
#     binary_ent += P[i] * np.log(Q1[i]) + (1 - P[i]) * np.log(1 - Q1[i])
# print(-binary_ent)  # 0.6851790109107685
binary_ent1 = - P * np.log(Q1) - (1 - P) * np.log(1 - Q1)
binary_ent2 = - P * np.log(Q2) - (1 - P) * np.log(1 - Q2)

print(binary_ent1.sum() / P.size)  # 0.22839300363692283
print(binary_ent2.sum() / P.size)  # 1.5701769005486392
# np.log 为自然对数

print(log_loss(P, Q1))
print(log_loss(P, Q2))
# 0.22839300363692283
# 1.5701769005486392

从上述结果中可以看出预测值$Q1$的交叉熵明显比$Q2$的小，说明$Q1$的预测更接近期望值。

对于多分类来说，其交叉熵计算方式如下
$$H(P,Q)=-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^{M}P(x_{i,j})\log (Q(x_{i,j}\left)\right)$$
among them,$P(x_{i,j})$ means sample$x_i$The true class label is $j$ is$1$ , otherwise$0$；$Q(x_{i,j})$ means sample$x_i$predicted as the jjthProbability of class$j$$N$ is the sample size,$M$ is the number of class labels.

Suppose there are $3$ samples, a total of$0\sim 9$ out of$10class$ labels,$P$ represents the real label of the sample, which is a vector;$Q$ represents the value of the model's category prediction for each sample, which is a$3$ lines$A\; matrix\; of\; 10$ columns, each row represents a pair of samplesiiA vector consisting of category prediction probability values ​​of$i\; ,\; to\; calculate\; the\; cross\; entropy.$

import numpy as np
from sklearn.metrics import log_loss
from sklearn.preprocessing import LabelBinarizer

P = np.array(['1', '5', '9'])  # 样本的真实标签

Q = np.array([
    [0.1, 0.7, 0, 0, 0.01, 0.19, 0, 0, 0, 0],
    [0, 0.1, 0, 0.1, 0, 0.8, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0.1, 0, 0.9]
])  # Q[i]是一个向量 Q[i][j]表示P[i]被预测为j类的概率

labels = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']  # 标签集合
sklearn_multi_ent = log_loss(P, Q, labels=labels)
print(sklearn_multi_ent)  # 0.22839300363693002

# 根据公式计算
# 先对类别标签进行二值化为0&1  数字标签和非数字标签都需要
LB = LabelBinarizer()
LB.fit(labels)
bin_P = LB.transform(P)  # 真实标签进行转换
print(bin_P)
# [[0 1 0 0 0 0 0 0 0 0]
#  [0 0 0 0 0 1 0 0 0 0]
#  [0 0 0 0 0 0 0 0 0 1]]

N = P.size
M = len(labels)
eps = 1e-15  # 预测概率的控制值

multi_ent = 0.0
for i in range(N):
    for j in range(M):
        if Q[i, j] > 1 - eps:
            Q[i, j] = 1 - eps
        if Q[i, j] < eps:
            Q[i, j] = eps
        multi_ent += -bin_P[i, j] * np.log(Q[i, j])

print(multi_ent / N)  # 0.22839300363692283

Suppose $P$ is the expected (true) distribution,$Q$ is the model (actual) distribution,minimize the cross entropy$H(P,Q)$ can make the model distribution approximate the real distribution.

References

[1] "Deep Learning" flower book
[2] related github