UTAH STATE UNIVERSITY
ELECTRICAL ENGINEERING DEPARTMENT

EE 7680 Information Theory

BSS Programming Assignment

Part I

Entropy maximization

1.

Generate a scalar random variable x which is exponentially distributed. Let y = g(w x+ w₀). Develop an algorithm to train the weights w and w₀ so that the entropy of y is maximized (train using successive independent samples of x). Use both the standard gradient and also natural gradient. Do this when

$$g(u) = \frac{1}{1+e^{-u}}.$$ (1)

Plot the histogram of x and the histogram of y, and compare with the function g. Determine the entropy of y.

2.

Repeat the above when

$$g(u) = \tanh(u).$$ (2)

3.

Repeat the above when g(u) is the CDF of x.

4.

Repeat all of the above when x is uniformly distributed from [-2,2].

Part II

Blind source separation.

1.

Select at random a mixing matrix A.

2.

Generate 4 independent exponentially distributed exponentially distributed variables in an input vector $\sbf$, and form $\xbf = W \sbf$.

3.

Develop a training algorithm to separate $\xbf$ by $\ybf = g(W \xbf + \wbf_0)$, using g(u) from (1). Make a plot demonstrating the convergence of the algorithm of the form |W A - I| (taking into account the permutations!).

4.

Repeat using g(u) from (2). Use both standard gradient and natural gradient.

5.

Now gather four distinct audio sources, including at least one music source into .wav files. Load them in, and mix them using a randomly generated W matrix. Save the mixtures to .wav files Apply the training algorithms to them (different kinds of g and different kinds of gradient). Play the separated signals, and save the results to .wav files.

Part III

Write up.

1.

Turn in a disk with your .wav files.

2.

Turn in a written report including plots, code, and a description of what you learned.