[Easy-to-understand communication] Shannon's Information Theory: Channel Capacity and Typical Sets

Author: Venceslas
link: https: //www.zhihu.com/question/67066372/answer/251430955
Source: know almost
copyrighted by the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.

For the strict proof, please refer to the proof of theorem 7.7.1 of the channel coding theorem in Elements of Information Theory by Tomas M.Cover.

To illustrate, we first introduce the concept of a typical set. The typical set is that for the iid sequence X^n, the following probability relations are satisfied:

A collection of sequences. It can be seen from the above formula that when \epsilon tends to zero, its probability tends to be the same. And from the theorem of large numbers, we can choose any sequence, and the probability of getting a sequence in a typical set is:

That is, we take the sequence from all possible sequences, and always take the sequence in the typical concentration with a probability of one. And it can be seen that the number of sequences in a typical set is $2^{nH(X)}$ .

Below we explain why the channel capacity can be represented by the amount of mutual information. For discrete memoryless channels, the sending sequence is $X^n$ , subject to iid X~P(X), we only consider the sequence in the typical set (because the sequence is arbitrary, the probability of fetching the sequence in the typical set tends to 1), that is , the set of sending sequences The size is $2^{nH(X)}$ . Similarly, at the receiving end, we generate the receiving sequence according to the sending sequence and obey the conditional distribution P(Y|X), and the typical set size corresponding to this distribution is $2 ^ {nH (Y | X)}$ . And the typical set size based only on the $Y ^ n$ distribution is $2 ^ {nH (Y)}$ . In order to be able to recover the sending sequence based on the receiving sequence, we require that any two sending sequences cannot produce the same output sequence, otherwise we cannot recover the sending sequence from the receiving sequence. That is, the set of receiving sequences corresponding to any two sending sequences cannot overlap, so we can allow $2 ^ {nH (Y)} / 2 ^ {nH (Y | X)} = 2 ^ {nI (X, Y)}$ at most one sequence to pass the channel . This briefly explains why the amount of mutual information can be used to represent the channel capacity.

Note: $2 nI (X, Y)}$ transmitting end sequences, each sequence corresponding to the transmitting end up receiving terminal $2 ^ {nH (Y | X)}$ sequences were, as shown below, many. For noise-free channel $H(Y|X)=0$ , $I(X,Y)=H(Y)$ the sending

The basis of Cover's information theory is much clearer than I said. The key is to understand and accept the concept of typical sets. If the above description is not very clear, you can look at the description of the channel capacity under the AWGN channel (filling model, in the English version p324-325), which is very intuitive.

[Easy-to-understand communication] Shannon's Information Theory: Channel Capacity and Typical Sets

Guess you like