1. The world is uncertain, all functional expressions y=f(x) only exist in theory, and the information observed in the real world is information with randomness. Therefore, it is a reliable idea to summarize the rules from experience and use statistics and probability theory.
2. What is the cause of uncertainty?
1"The world itself is uncertain (such as quantum mechanics) 2"Unable to grasp all the factors that affect the result (in front of the creator, human beings are insignificant) 3"Incomplete modeling (discarding unnecessary details, too many details, is not conducive to application)
3. Conceptually, a functional relationship is a special case of a probabilistic relationship (the probability that a function value appears is always equal to 1). Therefore, the functional relationship can be transformed into a probability model (the core is that the sum of the probability is 1). If it can satisfy some better analytical properties (continuous, differentiable, integratable, convex functions) at the same time of transformation, it is very cool thing. In the name of the function, these probabilistic functions are generally called softXXX
4. Bayes' theorem: p(y|x) find a way to express it with p(x|y)
5. Information Theory: The occurrence of an unlikely event provides more information than the occurrence of a very probable event.
6. Cross-entropy, which describes the difference between the calculated distribution and the observed distribution.
7. Maximum relief estimation: Small-probability events are not easy to occur. Correspondingly, the sampled data can be considered as a high-probability event, and the probability of the sampled data can be maximized to determine the parameters of the probability distribution.