Anatomy of the Kalman family from scratch - (02) Derivation and understanding of Bayesian formula → Know why

Explaining the summary link of a series of articles about slam:The most comprehensive slam in history starts from scratch, for the Kalman family explained in this column Zero anatomy link:The Kalman family’s anatomy from scratch - (00) catalog latest explanation without blind spots: https://blog.csdn.net/weixin_43013761/article/details/133846882

${\color{blue}{The center directly below the end} provides my \color{red} contact information, \color {blue}Click on your photo to display WX→official certification}$

$\color{red} Solemnly declare: This series of blogs are exclusively owned by me (WenhaiZhu). Reprinting and plagiarism are prohibited. , thank you for the first report!$

I. Introduction

If we start with the derivation of the Kalman filter formula, I personally feel that this is not appropriate because it will cause too much confusion? For example, why do we need this thing and what is its use? Why do you have to use it? After all, you are not in school, unlike high school, just for exams. My personal purpose here is to thoroughly understand, think through, and use it. In other words, when others use this algorithm, they can easily understand their purpose, why they choose this algorithm, and what are the advantages of this algorithm. Advantage. For another example, when a problem occurs when using other people's open source algorithms for a project, if you know the reason, you can locate the cause faster, instead of just fishing blindly and relying on luck. Without further ado, let’s get started.

To explore Bayesian filtering, you have to mention Bayesian formula. Before deriving it in detail, let’s talk about some common things in daily life:
$\color{blue} (1) Roommate is late:$ When your roommate was late for class today (9.00 class), did you occasionally have a thought, what is the probability that he would get up before 8.30?
$\color{blue} (2) Cause of car accident:$ When you were passing by the sidewalk, you saw a car accident. I believe many people will wonder, was it caused by running a traffic light, or was it an accident while driving normally?
$\color{blue} (3) Children fighting:$ Your child got into a fight with a neighbor’s child. Your child came to complain and told you that he was bullied. You thought to yourself, whose child started the fight? ?
$\color{blue}(4) The detective solves the case:$ This example is quite typical. For example, a murder occurs and three suspects are identified. Who is the real murderer? Or who has the highest probability of being the murderer.

These things all have one thing in common, which is to deduce the cause from the result. If I have enough time, I will try to analyze the above case using mathematical formulas. I will not analyze some relatively simple forward derivation, such as a deck of cards, what is the probability that one card is randomly selected and it is the Ace of Hearts; what is the probability of a coin being tossed 10 times in a row and 3 heads? ; There are 10 cards in the box worth one dollar, 5 cards worth five dollars, one card worth ten dollars, and 3 cards are randomly selected. How much can you get? These are all things based on probability theory.

There is a course in the university called Probability Theory and Mathematical Statistics. In fact, Probability Theory and Mathematical Statistics are two closely related but different fields of mathematics. They are both used to study and understand uncertainty and random phenomenon. Here are their definitions and main concerns:

Probability Theory:
Probability theory is a branch of mathematics that studies mathematical models of uncertain events and random phenomena, as well as their probabilistic properties and laws.
The main focus includes the probability of events, random variables, probability distribution, conditional probability, independence, expected value, variance, etc.
Probability theory is used to describe and analyze the likelihood of an event occurring. It provides a framework to quantify uncertainty so that reasonable decisions and inferences can be made.

Mathematical Statistics:
Mathematical statistics is a branch of mathematics concerned with collecting and analyzing data to make inferences and decisions about general characteristics.
The main focus includes parameter estimation, hypothesis testing, confidence intervals, variance analysis, regression analysis, etc.
Mathematical statistics helps us extract information from collected data, understand population characteristics, and answer questions about the population through statistical inference.

$\color{red}Personal understanding:$ Generally, what is related to forward derivation (cause → effect) is more closely related to probability theory; what is related to reverse derivation (effect → cause) is related to Mathematical statistics is more closely related.

Before explaining Bayesian filtering, you need to understand the Bayesian formula. For the derivation process of this formula, please seeDissection of the Kalman family from scratch - (01) Preliminary knowledge points , here are the Bayesian formulas for discrete and continuous random variables:

$\color{blue}(1)Discrete:$
$\color{Green} \tag{01} P\left(X_{i} \mid Y\right)=\frac{P\left(Y \mid X_{i}\right) P\left(X_{i}\right)}{P(Y)}=\frac{P\left(Y \mid X_{i}\right) P\left(X_{i}\right)}{\sum_{i=1}^{n} P\left(Y \mid X_{i}\right) P\left(X_{i}\right)}$
$\color{blue}(2) Connection:$
$\color{Green} \tag{02} f_{X \mid Y}(x \mid y)=\frac{f_{X, Y}(x, y)}{f_{Y}(y)}=\frac{f_{Y \mid X}(y \mid x) f_{X}(x)}{\int_{-\infty}^{+\infty} f_{Y \mid X}(y \mid x) f_{X}(x) \mathrm{d} x}=\eta f_{Y \mid X}(y \mid x) f_{X}(x)$ In order to deeply understand the role of Bayesian formula, let’s cite a few Examples are used to deepen the impact, so that the formula is derived but cannot be used.

2. Examples of Bayesian formula

Example 1 (discrete):

Based on your own child’s understanding (subjective experience), if there is a fight with other children, the probability of him striking first is assumed to be $P (主动) = 0.2$ , the probability of taking action later is $P (被动) = 0.8$ , in addition, the probability of taking the initiative to fight and telling me is $P (Adventure ∣ main action) = 0.3$ , the probability of taking the initiative to fight and not informing me is $P (Unannounced ∣ Main activity) = 0.7$ ; Of course, it is also possible that $P (告之 ∣ 被动) = 0.9$ , $P (not notified ∣ affected a>) = 0.1$ . For convenience, the conditions are listed together: $\\P(Tell | Active)=0.3 ~~~~~~~~~ P(Don’t tell | Active)=0.7 \\P(Tell | Passive)=0.9~~~~~~~~P(Don’t tell | Passive)=0.1$ Having known the above conditions, ask the baby what is the probability of taking the initiative to fight today? According to the Bayesian formula, it is expanded as follows: $(Active|Tell)=\frac{P(Tell|Active)P(Active)}{P(Tell|Active)P(Active )+P(Tell | Passive)P(Passive)}=\frac{0.06}{0.06+0.72}=0.077$ The above is the result we require. If you doubt whether you are correct, you can calculate it $P (affected ∣ announcement$ Approximate probability, given $)$ $P (Take the initiative ∣ Inform$ $P(passive|report)=\frac{P(report|passive)P(passive)}{P(report|passive)P(passive)+P(report of|Active)P(Active)}=\frac{0.72}{0.72+0.06}=0.923$ If the addition result is 1, it means it is correct. The calculation is as follows: $)$
$P (被动 ∣ 告之) = \frac{P ( 告之 ∣ 被动 ) 被动 ( P )}{P ( 告之 ∣ 被动 ) 被动 ( P ) + P ( Adventure ∣ main action ) main action ( P )} = \frac{0.72}{0.72 + 0.06} = 0.923$ So the above calculation process is correct. Generally speaking, the probability of the baby taking the initiative to fight today is still very low. A question that needs to be considered deeply here is that the probability of a baby actively fighting was originally 0.2, but why it was later reduced to 0.077 after calculation. This is because with the prior knowledge of the baby's personality, if the baby takes the initiative to fight, there is a high probability that he will not tell himself. This revised probability $P (Adventure ∣ main action$ It is also a corrected estimate. $)$

Example 2 (discrete):

Assume that 10% of the people in the world do not like to drink yogurt, and 90% of the people like to drink yogurt. The probability of people who like to drink yogurt admit that they like to drink yogurt is 99%, and the probability of people who do not like to drink yogurt to say that they like to drink yogurt is 99%. is 2%. If you interview someone, this person says that he likes to drink yogurt. Ask this person what is the general idea that he likes to drink yogurt?

Answer: The probability that people who like to drink yogurt say that they like to drink yogurt is 90% * 99%, and the probability that people who don’t like to drink yogurt say that they like to drink yogurt is 10% * 2%. So the probability that this person really likes to drink yogurt is: $\frac{90 \% \times 99 \% }{90 \% \times 99 \%+10 \% \times 2 \%}=99.78 \%$

Example 3 (continuous):

The previous two examples were both discrete and did not use the probability density function (PDF). Now let's give a continuous example using distance measurement. Assuming that the renovated house needs to be inspected and accepted, the length of the living room needs to be measured to see if it meets the standards. According to the drawings given by the sales unit, the living room is 900cm long. If the deviation is within ±1cm, it meets the acceptance criteria. Then we buy a tape measure online. According to official information, the accuracy of the tape measure is plus or minus ±2cm (no matter how long the distance is measured). In addition to accuracy, the manufacturer does not provide any extra parameters.

Then let’s discuss the data given by the sales unit, which is 900cm, and the deviation is within ±1cm. Is this worth trusting? If it is credible, we have no need to calculate it, so we are skeptical here, at least not 100% trustworthy. Therefore, we believe that the data conforms to a normal distribution probability density function, as follows;
$\color{Green} \tag{03} f(x)=\frac {1}{ \sigma \sqrt{2 \pi}} e^{-\frac{(x-\mu)^{2}}{2 \sigma^{2}}}~~~~~~~~ ~~~X \sim N\left(\mu, \sigma^{2}\right)$ onward $\mu$ Display uniformity, $\sigma$ The length of the living room given by the selling unit is an a priori value, here $in = 900$ cm, standard difference $\sigma=1$ cm. Then we took the tape measure to measure, and the measurement result was recorded as $897$ cm, this is an observed value. So what do we actually want to get? In fact, the probability of the house symbol acceptance standard. If the probability is high, the symbol is considered standard. According to the Bayesian probability density formula of Bayesian continuous random variables, we can get:
$~|~Y=897)=\int f_{X|Y}(x|y) \mathrm dx=\int \eta f_{Y|X}(y|x)f_X(x) \mathrm dx$ According to the above formula, it can be seen that integration is required, where < /span> $f_X(x)$ represents the prior value, which has been assumed to conform to the normal distribution of. For $f_{Y|X}(y|x)$ represents the likelihood probability density function. In this example, it means that when $X = 900$ time, $AND = The probability density function of 897$ , based on the sensor accuracy information, is also assumed to conform to the normal distribution, $in = 897$ cm, standard difference $\sigma=2$ cm. Then the following two positive probability density functions are currently known: $1^{2}\right) \\Likelihood probability density function: ~f_{Y|X}(y|x)=Y \sim N\left(897, 2^{2}\right)$ Specify the slightly larger, slightly larger range: $\color{Green} \tag{06} \eta f_{Y|X}(y|x)f_X (x)=\eta \frac{1}{ 1 \sqrt{2\pi}} e^{-\frac{(x-900)^{2}}{2 *1^{2}}}*\ frac{1}{2\sqrt{2\pi}} e^{-\frac{(x-\mu)^{2}}{2*2^{2}}}=\eta \frac{1} { 0.8 \sqrt{2\pi} } e^{-{\frac{(x-899.4)^{2}}{2*{0.8}^{2}}}}$ Let’s ignore the calculation process of the product of the two normal distributions in the above formula. The blog will conduct detailed derivation, just look at the results. By observing the above formula, we can know that the posterior probability probability density function is $\eta *N\left(899.4, 0.8^{2}\right )$ , first of all, the value that can be seen intuitively, relative to the prior probability density function $N\left(900, 1^{2}\right)$ The standard deviation has been reduced, which means the accuracy has been improved. Generally speaking, the actual living room has a high probability of 899.4cm , the deviation is about 0.8cm, here $\eta$ is a constant.

Although we have reached this step, it still seems that our question has not been answered, that is, whether the actual living room meets the acceptance standards, and the integral of formula (23) has no upper and lower limits, although we know $\eta=1/{\int_{-\infty}^{+\infty} f_{Y \mid X}(y \mid x) f_{X}(x) \mathrm{d} x}$ . We then analyze that according to the sales department’s instructions, 899.0~991.0 is a normal compliance standard, so we will calculate the probability that the length of the living room is greater than or equal to 899.0, that is: $P (899.0 \leq X ∣ Y = 900)$ . Essentially, the posterior probability probability density function is to find $\eta*N\left(899.4, 0.8^{2}\right)$ $x\in[899.0,\infty ]$ . For example, find the probability $P$ After that, if it is greater than 80%, then we can think that the developer’s feasibility is relatively high, which means it meets the acceptance criteria. I have omitted the calculation process here. Divide by here $\eta$ , its function is normalized, which is actually $-\infty≤X+\infty|Y=900)$ . To put it bluntly, it is $\left(899.4, 0.8^{2}\right)$ are not consistent between 0 and 1, so normalization is performed.

3. Bayesian formula of normal distribution

The above example of continuous random variable and normal distribution, Example 3, can be directly substituted into the following formula to obtain the result:
$\color{Green} \tag{07} Prior probability density function: ~f_X(x) =X \sim N\left(\mu_1, \sigma_1^{2}\right)=\frac{1}{ \sigma_1 \sqrt{2 \pi}} e^{-\frac{(x-\mu_1) ^{2}}{2 \sigma_1^{2}}}$ $~f_{Y|X}(y|x)=X \sim N\left(\mu_2, \sigma_2^{2}\right)=\frac{1}{\sigma_2\sqrt{2\pi}} e ^{-\frac{(x-\mu_2)^{2}}{2\sigma_2^{2}}}$ $\color{Green} \tag{09} f_{X \mid Y}(x \mid y)=\frac{f_{X, Y}(x, y)}{f_{Y}(y)}=\frac{f_{Y \mid X}(y \mid x) f_{X}(x)}{\int_{-\infty}^{+\infty} f_{Y \mid X}(y \mid x) f_{X}(x) \mathrm{d} x}$ $\color{Green } \tag{10} also define the boundary value:~f_{X|Y}(x|y)=N\left(\frac{\sigma_{1}^{2}}{\sigma_{1}^2 +\sigma_{2}^{2}} \mu_{2}+\frac{\sigma_{2}^{2}}{\sigma_{1}^{2}+\sigma_{2}^{2} } \mu_{1} , \frac{\sigma_{1}^{2} \sigma_{2}^{2}}{\sigma_{1}^{2}+\sigma_{2}^{2}} \right)$ From this formula, it can be clearly known that after the likelihood probability density function After correction, the standard deviation (error) of the prior probability density function can be reduced. Because $\frac{\sigma_{1}^{2} \sigma_{2}^{2}}{\sigma_{1 }^{2}+\sigma_{2}^{2}}$ must be less than $\sigma_{1}^{2}$ 与 $\sigma_{1}^{2}$ of. It is more troublesome to directly expand the calculation. You can use the mathematical tool Mathematica to directly output the results. If there is sufficient time in the future, we may expand the length and conduct theoretical derivation.

4. Bayesian filtering idea

In order to make it easier for everyone to read my series of articles, some mathematical symbols need to be specified here to avoid accidental confusion when seeing the symbols later. First make some introductions or agreements. At the same time, the derivation of important formulas will also be recordedAnatomy of the Kalman family from scratch - (01) Preliminary knowledge points In this article, I personally recommend that after reading it, Go back and read that blog post.

$\color{red}Core (personal understanding):$ First of all, Bayesian filtering is not a specific implementation of a specific algorithm, but an ideological guidance, similar to the abstract class in C++ programming, which only defines some virtual functions. Here is a more appropriate example, such as primary school Chinese class, children will learn strokes, one horizontal, one vertical, one stroke, etc. These are similar to Bayesian filtering, which is relatively basic and abstract. However, through the combination of strokes, various Chinese sub-pictures can be realized, such as you, me, him, good, bad, etc. These actual words are like Kalman filter, particle filter, etc. Although every child writes a word, as long as it is written (combination of strokes) correctly, then it is a correct word, but what? Every child's handwriting is different, some are good-looking, some are not good-looking, some are big, some are small, but these words are all the children's own characteristics. It's like here, all walks of life and different algorithms may all use Kalman filtering, but they are not the same, and there will be some differences. Therefore, when we use these instantiated algorithms, we also need to make some changes or adjustments based on the actual situation.

For example, in the above three examples, the assumed known conditions and Gaussian distribution have actually been instantiated, because Bayes is given $P\left(Y \mid X_{i}\right)$ 与 $P\left(X_{i}\right)$ , but it does not give specific values. The values are our assumptions. In actual application, in order to obtain these probability values, modeling is usually required. For example, if Gaussian distribution is used for modeling, it can be called Gaussian filtering. So what is filtering? In fact, it is easy to understand in one sentence. The function of filtering is to reduce noise. For example, if you buy a thermometer, the parameters purchased have stated that the accuracy deviation of the sensor is ±1°, then this ±1° It's noise, which is determined by the manufacturer after various tests. Then, through the filtering algorithm, the accuracy can be improved, if it reaches a deviation of ±0.5°. In this way, the noise is obviously reduced.

5. Summary

Through the explanation of this blog, I mainly learned the following concepts: first verification probability, likelihood probability, and posterior probability. In the subsequent process, in order to be consistent with various textbooks or blogs, it should be mentioned here that the prior is also called prediction, and the likelihood is also called observation. They may be mixed in the subsequent process. I hope everyone will notice in advance to avoid more doubts. About:
$\mid Y}(x \mid y)=\frac{f_{X, Y}(x, y)}{f_{Y}(y)}=\frac{f_{Y \mid X}(y \mid x) f_{X}(x)}{\int_{-\infty}^{+\infty} f_{Y \mid X}(y \mid x) f_{X}(x) \mathrm{d } x}=\eta f_{Y \mid X}(y \mid x) f_{X}(x)$ is a very important formula. Although it is abstract, the subsequent Many derivations are based on this abstract formula, or the concretization and instantiation of this formula. There are thousands of instantiation methods, but it doesn’t matter. We will analyze and explain them one by one later.

This blog explains an example of Bayesian filtering of continuous random variables using the normal distribution modeling method. However, strictly speaking, this example is not comprehensive. In fact, Bayesian filtering usually changes along the time axis. There are multiple prior values and predicted values. And there is a recursive relationship. The specific details will be analyzed in detail in the next blog.