Author: Yi Zhen
link: https: //www.zhihu.com/question/22178202/answer/161732605
Source: know almost
copyrighted by the author. Commercial reprint please contact the author authorized, non-commercial reprint please indicate the source.
link: https: //www.zhihu.com/question/22178202/answer/161732605
Source: know almost
copyrighted by the author. Commercial reprint please contact the author authorized, non-commercial reprint please indicate the source.
The following is my understanding step by step leads to sources of information and entropy formula:
Information entropy formula
First throw entropy formula is as follows:
Which represents a random event X is
the probability of the following formula to gradually introduce sources of information entropy!
Information
The amount of information is a measure of information, just as a measure of time in seconds, when we consider a discrete random variable x, when we observe a specific value to this variable, how much information we receive it?
How much information is used to measure the amount of information, we received information about the occurrence of specific events with.
With size information about the probability of random events. The smaller the probability of happening of the greater amount of information generated , such as Hunan generated by the earthquake; the greater the probability of happening the amount of information generated by the smaller , (such as the sun rose from the east of it must have happened Well, no amount of information ). This is well understood!
example
Brain mend our daily conversation:
Brother came up to me, standing wave ah, you Hunan, a major earthquake today.
Me: ah, it is not possible, so the heavyweight news! Multi Hunan low probability of an earthquake ah! Brother, you told me about it, a huge amount of information , I immediately called to ask parents what was happening.
Here comes a junior sister apprentice: standing wave brother, I found a significant amount of intelligence, the original amount of the Tokugawa brothers have a girlfriend - a year into the early Tokugawa laboratory than junior sister apprentice, full laboratory students know about it. I laugh: Ha ha ha ha, this thing we all know, is not gold, the next time some other valuable news gossip it! orz, escape ~
Therefore, the amount of information a specific event should be as it diminishes the probability of occurrence, and can not be negative.
But this represents how to find information in the form of a function of it?
Much as the probability increases to reduce the functional form! Do not worry, we have the following of this nature
If we have two unrelated events x and y, then the information obtained during the same time we observed the occurrence of two events should be equal to the observed information obtained when the respective events occur and, namely:
h(x,y) = h(x) + h(y)
As the x, y are two unrelated events, satisfying p (x, y) = p (x) * p (y).
According to the above derivation, we can easily see that h (x) and a certain number of p (x) about (because after multiplied by the number really is only logarithmic form, can, you can try to deal with the form of the sum of the number) . Therefore, the amount of information we have the following formula:
The following two solve a question?
(1) why there is a negative sign
Among them, the negative sign in order to ensure that the information must be positive or zero, it can not be negative!
(2) Why base-2
This is because the amount of information we need only meet the low probability events x corresponds to the high amount of information. So the choice of the number is arbitrary. We just follow the prevailing traditional information theory, use 2 as a substrate for a number of!
Entropy
Here we formally elicit information entropy.
Is a measure of the amount of information specific event information brings, and entropy is expected to come out before the results of the amount of information possible - consider all the possible values of the random variable, that is, all possible events It brings the amount of information desired. which is
Convert it to:
The final source of our formula is derived completed.
Here I say a understanding of entropy. Information entropy can also be used as a measure of the complexity of the system, if the system is more complex, it appears more types of different situations, so his information entropy is relatively large.
If a system more simple, the kind of situation occurs rarely (in extreme cases as Case 1, the corresponding probability is 1, then the corresponding information entropy is zero), then the information entropy is small.
This is all the information entropy idea I understand, I hope everyone is pointing the wrong exchange. We hope to help everyone understand ~