Next: The channel coding theorem Up: lecture7 Previous: A closer look at

Jointly typical sequences

Recall the definition of a typical sequence: it is the sequence we expect (probabilistically) to occur, given the statistics of the source. We had a theorem regarding the approximate number of typical sequences, the probability of a typical sequence, and so forth. In this section we generalize this.

All along this semester we have used the notion of representing sequences of random variables as vectors. For example, the r.v.'s (X₁,X₂) we could represent with a vector-valued random variable X. In a sense, this is all that we are doing with the jointly-typical sequences. We consider sequences (xⁿ,yⁿ) as if there were simply some sequence zⁿ, and ask: for the set of sequences zⁿ, what are the typical sequences:
$\begin{definition} The set $A_\epsilon^{(n)}$\ of {\bf jointly typical } sequen... ...Y)\right\vert < \epsilon\right\} \end{aligned}\end{displaymath}\end{definition}$
From this definition, we can conclude similar sorts of things about jointly typical sequences as we can about typical sequences:
$\begin{theorem} (Joint AEP). \begin{enumerate} \item $\text{Pr}((X^n,Y^n) \in... ...epsilon) 2^{-n(I(X;Y)+3\epsilon)}. \end{displaymath}\end{enumerate}\end{theorem}$

$\begin{proof} By WLLN, \begin{displaymath}-\frac{1}{n}\log p(X^n) \rightarrow -... ...e sum). Similar arguments can be used to derive the last inequality. \end{proof}$

There are about 2^nH(X) typical X sequences, about 2^nH(Y) typical Y sequences, and only about 2^nH(X,Y) jointly typical sequences. This means that if we choose a typical X sequence and independently choose a typical Y sequence (without regard to the X sequence), in not all cases will the sequence (Xⁿ,Yⁿ) so chosen be jointly typical. In fact, from the last part of the theorem, the probability that the sequences chosen independently will be jointly typical is about 2^-nI(X;Y). This means that we would have to try (at random) about 2^nI(X;Y) sequence pairs before we choose a jointly typical pair. Thinking now in terms of fixing Y and choosing X at random, this suggests that there are about 2^nI(X;Y) distinguishable sequences in X.

Next: The channel coding theorem Up: lecture7 Previous: A closer look at

Todd Moon
2000-03-22