Question about mutual information

Question about mutual information

by Wenxin Pan -
Number of replies: 2

Hello, I have a question about mutual information. My question in short is:
Does this formula hold?: I(X;Y=y)=H(X)-H(X|Y=y).

I just follow the example 2.6.1 in Cover's book. (The picture below)

By definition, I(X;Y=1) should be calculated as I(X;Y=1)=\sum_{x,y=1}p(x,y=1)log_2(\frac{p(x,y=1)}{p(x)p(y=1)})=0+\frac{3}{4}log_2(\frac{3/4}{7/8*3/4})=0.144 bit \)

If by the formula above, I(X;Y=1)=H(X)-H(X|Y=1)=H(X)=0.544 bit, which is different from 0.144 bit. So I think this formula does not hold. But why is this? It looks quite right.

Thank you.



In reply to Wenxin Pan

Re: Question about mutual information

by Wenxin Pan -

I might write the long formula again:

I(X;Y=1)=\sum_{x,y=1}p(x,y=1)log_2(\frac{p(x,y=1)}{p(x)p(y=1)})=0+\frac{3}{4}log_2(\frac{3/4}{7/8*3/4})=0.144

In reply to Wenxin Pan

Re: Question about mutual information

by Thomas Weinberger -
Dear Wenxin,

The issue with your calculations is that mutual information is only defined between two random variables (side note: there also exist extensions of MI for >2 RVs, and/or conditioned on RVs).

You are however trying to calculate the mutual information between a random variable X and an event \{Y=1\}, which is ill-defined. Moreover, what you actually calculate on the right-hand side of your reply is the mutual information between X and a constant random variable Y which is always 1. Obviously this is not the same Y as defined in your table above, hence you get inconsistent results.

Hope this helps.

Best,
Thomas