【Artificial Intelligence】—Uncertainty, prior probability/posterior probability, probability density, Bayes rule, naive Bayes, maximum likelihood estimation

Uncertainty

Uncertainty and rational decision making

basic probability notation

Prior probability (unconditional probability) / posterior probability (conditional probability)

Random Variables

Probability Density

joint probability distribution

axiom

complete joint distribution

Probability calculus

independence

bayes rule

Example 1

You have two envelopes to choose from. One envelope contains a red ball (worth $100) and one black ball, and the other envelope contains two black balls (worth zero).

You choose an envelope at random, then draw a ball from that envelope at random, and it turns out to be black.

At this point, you can choose whether to change to another envelope. The question is, should you switch or not?

---

E: envelope, 1 means an envelope with a red ball, 2 means an envelope with black balls $1=(R,B),2=(B,B)$

B: the event of drawing a black ball

$$\begin{gathered} Bayes\text{'}\; rule: \\ P(E|B)=\frac{P(B|E)P(E)}{P(B)} \end{gathered}$$

We want to compare $Compare:P(E=1|B)$ vs. $P(E=2|B)$

$$\begin{gathered} Get\; the\; black\; ball\; in\; the\; red\; ball\; envelope:P(B|E=1)=0.5 \\ Get\; the\; black\; ball\; in\; the\; black\; ball\; envelope:P(B|E=2)=1 \end{gathered}$$

$$The\; probability\; of\; gettingenvelopes1and2\; is\; the\; same:P(E=1)=P(E=2)=0.5$$

$$\begin{array}{rl}& Probability\; of\; drawing\; a\; black\; ball:\\ & The\; marginal\; probability\; ofBon\; the\; value\; ofE\\ P(B)& =P(B|E=1)P(E=1)+P(B|E=2)P(E=2)\\ & =(0.5)(0.5)+(1)(0.5)\\ & =0.75\end{array}$$

$$\begin{gathered} Having\; drawn\; a\; black\; ball,\; the\; probability\; that\; this\; envelope\; is\; a\; red\; ball\; envelope: \\ P(E=1|B)=\frac{P(B|E=1)P(E=1)}{P(B)}=\frac{(0.5)(0.5)}{(0.75)}=\frac{1}{3} \end{gathered}$$

$$\begin{gathered} A\; black\; ball\; has\; been\; drawn,\; the\; probability\; that\; this\; envelope\; is\; a\; black\; ball\; envelope: \\ P(E=2|B)=\frac{P(B|E=2)P(E=2)}{P(B)}=\frac{(1)(0.5)}{(0.75)}=\frac{2}{3} \end{gathered}$$
$$\begin{gathered} By\; calculation,\; the\; probability\; that\; the\; envelope\; is1\; after\; drawing\; a\; black\; ballis1/3, \\ The\; probability\; that\; the\; envelope\; is2is2/3.Therefore,\; changing\; the\; envelope\; can\; increase\; the\; probability\; of\; getting\; a\; red\; ball. \end{gathered}$$

---

Example 2

A doctor performs a test that is 99% reliable in that 99% of sick people test positive and 99% of healthy people test negative. The doctor estimated that 1% of the entire population was sick.
So, for a patient who tests positive, what is the probability that he is sick?

---

We can use Bayes' theorem to calculate the conditional probability that a patient is sick. Let an event S represent a patient being ill and an event T represent a positive test result. Then the conditional probability sought is:

$$P(S|T)=\frac{P(T|S)P(S)}{P(T)}$$

Among them, $P(T|S)$ represents the probability that the test result is positive under the condition that the patient is sick,$P\left(S\right)$ representsthe prior probability,$P\left(T\right)$ represents the probability that the test result is positive.

$$\begin{gathered} According\; to\; the\; data\; given\; in\; the\; title,\; we\; have: \\ P(T|S)=0.99 \end{gathered}$$
$$P(S)=0.01$$
$$P(T)=P(T|S)P(S)+P(T|\overline{S})P(\overline{S})$$

$$in,\overline{S}Indicates\; that\; the\; patient\; is\; not\; sick.$$

$$\begin{gathered} According\; to\; the\; reliability\; of\; the\; test,\; we\; can\; get \\ P(T|\overline{S})=1-P(T|S)=0.01 \\ therefore \\ \begin{array}{rl}P(T)& =P(T|S\left)P\right(S)+P(T|\overline{S})P\left(\overline{S}\right)\\ & =(0.99)(0.01)+(0.01)(0.99)\\ & =0.0198\end{array} \end{gathered}$$

$$\begin{gathered} Substituting\; Bayesian\; formula,\; we\; can\; calculate\; the\; conditional\; probability\; of\; the\; patient\; being\; sick: \\ P(S|T)=\frac{(0.99)(0.01)}{0.0198}\approx 0.50 \\ Therefore,\; the\; probability\; that\; a\; patient\; who\; tests\; positive\; is\; sick\; is\; about50 \end{gathered}$$

---

Using Bayes' Rule: Combining Evidence

Naive Bayes

maximum likelihood estimation

Maximum Likelihood Estimation (MLE for short) is a commonly used parameter estimation method, which is used to estimate the parameters of the model based on known sample data. Its core idea is to select the parameter that can maximize the probability of the observed data as the estimated value.

Specifically, in maximum likelihood estimation, we assume that the sample data come from a certain probability distribution, but the parameters of the distribution are unknown. Our goal is to estimate these parameters from the sample data such that the distribution best explains the observed data.

Suppose we have a sample set $X=x_1,x_2,...,x_n$, each sample comes from a certain distribution f ( x ∣ θ ) f(x|\theta)Observations of$f$$($$x$$|$$\theta$$)$$\theta$ is a parameter of the distribution. We want to find the sample setXXThe joint probability density function L of$X$$L(X|\theta )$ parameter value$\theta$ . This joint probability density function can be expressed as:

$$L(X|\theta )=\prod _{i=1}^{n}f(x_i|\theta )$$

Our goal is to find the ability to maximize $L(X|\theta )$ 的 $Theta$ value. Therefore, the calculation of maximum likelihood estimation can be expressed as:

$${\hat{i}}_{MLE}=\arg \underset{i}{\max }L(X|\theta )$$

Sometimes we need to take the logarithm of the above formula to avoid computer calculation underflow, and the obtained formula is:

$${\hat{i}}_{MLE}=\arg \underset{i}{\max }\log L(X|\theta )=\arg \underset{i}{\max }\sum _{i=1}^n\log f(x_i|\theta )$$

The maximum likelihood estimation method is a commonly used parameter estimation method, which has the advantages of simple calculation and good theoretical basis. It has been widely used in statistics, machine learning, signal processing and other fields.

summary

The following is a compilation of important formulas in probability theory:

---

1. Conditional probability formula:

For event A and event B, the conditional probability is expressed as $P(A|B)$ represents the probability of event A occurring under the condition that event B occurs. The conditional probability formula is:

$$P(A|B)=\frac{P(A,B)}{P(B)}$$

---

2. Multiplication rule formula:

For event A and event B, the joint probability is expressed as $P(A,B)$ , which represents the probability that event A and event B occur at the same time. The multiplication rule formula is:

$$P(A,B)=P(A|B)P(B)$$

---

3. Chain rule formula:

For multiple events $A,B,C,D$ , whose joint probability is expressed as$P(A,B,C,D)$ , the chain rule formula can be expressed as:

$$P(A,B,C,D)=P(A|B,C,D)P(B|C,D)P(C|D)P(D)$$

---

4. Conditional chain rule formula:

For event A and event B, the joint probability is expressed as $P(A,B)$ , the conditional chain rule formula can be expressed as:

$$P(A,B|C)=P(A|B,C)P(B|C)$$

---

5. Bayes' theorem formula:

Bayesian theorem is a method of calculating posterior probability based on prior probability and conditional probability, which can be used for classification, prediction and other tasks. The formula for Bayes' theorem is:

$$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$$

---

6. Conditional Bayes' theorem formula:

For event A and event B, the conditional Bayes theorem formula can be expressed as:

$$P(A|B,C)=\frac{P(B|A,C)P(A|C)}{P(B|C)}$$

---

7. Additive/conditional probability formula:

For event A and event B, the additive/conditional probability formula can be expressed as:

$$P(A)=P(A,B)+P(A,\neg B)=P(A|B)P(B)+P(A|\neg B)P(\neg B)$$

---

These formulas are very important in probability theory and can be applied to problems in various fields such as statistics, machine learning, signal processing, financial fields, and medical fields. Proficiency in these formulas can help us better understand and solve practical problems.