Learning in Information - A First Look at the Door

Chapter 1 Content

Information entropy: a measure of the uncertainty of the information selection of the source
Signal: the carrier of the message, representing the physical quantity of the
message Source encoder : data compression, improve effectiveness
Channel coding : add error correction elements, improve reliability, anti-interference Ability Enhancement
Variance Formula :
D(X)=E{[xE(X) ² ]}=E(X ² )-E ² (X)

Chapter 2 Content

Self-Information: I(x _i )=-log p(x _i )
The definition of self-information reflects the uncertainty of time itself

Small-probability events contain large uncertainties and have a large amount of self-information, while high-probability events are the opposite.

Joint self-information:
I(x _i y _j )=-log p(x _i y _j )
If the two events are independent, then I(x _i y _j )=I(x _i )+I(y _j )

Conditional Self-Information: Conditional Self-Information I(x _i |y _j )=-log p(x _i |y _j )
I(x _i y _j )=I( _xi |y under a given condition of an event _j )+I(y _j )
I(x ₁ x ₂ …x _n )=I(x ₁ )+I(x ₂ |x ₁ )+…I(x _n |x ₁ x ₂ …x _n-1 )
That is, the occurrence of x _i will help reduce or increase the uncertainty of y _j
itself. Vera graph

Mutual information: the occurrence of event one gives the amount of information about event two
$$I(x_i;y_j)=\frac{p(x_i|y_j)}{p(x_i)}=log\frac{1}{p\left(x_i\right)}-log\frac{1}{p(x_i|y_j)}=I\left(x_i\right)-I(x_i|y_j)$$ indicates the uncertainty_i
when y_joccurs
The amount of mutual information can be positive or negative

The average self-information of a discrete set, also known as information entropy
$$H(X)=-\sum _{i=1}^{q}p(x_i)logp(x_i)$$

Entropy is symmetrical, the order of each component is changed, and the entropy is constant.
Entropy is non-negative, and the entropy of the determined field is the smallest. If it is 0,
entropy has scalability, and the increase of a minimal probability event, the entropy value can be regarded as constant.
Entropy has an additive value. Properties: H(XY)=H(X)+H(Y|X)
entropy has extreme value, maximum entropy theorem, the entropy of isoprobable field is the largest, that is, H(X)=log n
entropy has upward convexity, H _n (p ₁ ,p ₂ ,…p _q ) is a strictly upward convex function of the probability distribution (p ₁ ,p ₂ ,…p _{q )}