Information Theory: Self-Information, Entropy, Relative entropy,Cross entropy, Conditinal Entropy

Self-Information: $I(x) = \log \frac{1}{P(x)}$

Entropy: the average information
Entropy : $H(X)=E[I(X)] =E(\log \frac{1}{P(X)})=\sum_{x \in X} P(x)\log \frac{1}{P(x)}$
Relative entropy: $D(P || Q) = \sum_{x \in X} P(x)\log \frac{P(x)}{Q(x)}$
KL-divergence $D(P || Q)$ = Relative entropy
Cross entropy: $H(P || Q) = \sum_{x \in X} P(x)\log \frac{1}{Q(x)}$

Minimizing Relative entropy is equivalent to minize Cross Entropy.

Conditional Entropy : How much uncertainty left given the other.
First recall: Conditional Expectation
$E(Y|X=x)$ is fixed number
$E(Y|X)$ is a random variable of X

$H(Y|X=x)$ is analogous to conditional expectation taken the value on $x$
$H(Y|X)$ is a little bit different than the above conditional expectation. Here, we take another expectation on X. so that:
$H(Y|X)=\sum_{x \in X}p(x)H(Y|X=x)$
$=-\sum_{x \in X}p(x)\sum_{y \in Y}p(y|x)\log p(y|x)$
$=-\sum_{x \in X}\sum_{y \in Y}p(x)p(y|x)\log p(y|x)$
$=-\sum_{x \in X}\sum_{y \in Y}p(x,y)\log p(y|x)$
$=-\sum_{x \in X}\sum_{y \in Y}p(x,y)\log \frac{p(x,y)}{p(x)}$
$=\sum_{x \in X}\sum_{y \in Y}p(x,y)\log \frac{p(x)}{p(x,y)}$

Relation Between Joint and Conditional Entropy

H(X, Y) = H(X) + H(Y|X)

Mutual Information
$I(X;Y) = I(Y;X)$ by symmetric
$I(X;Y ) = H(X)-H(X|Y)$-the reduction of uncertain of X due to knowledge of Y
$=H(Y)-H(Y|X)$-The reduction of uncertain of Y due to knowldge of X