Ilya: 1 revision imported

2018-07-04T16:28:41Z

1 revision imported

nemenman>Ilya at 02:07, 3 October 2011

2011-10-03T02:07:47Z

New page

{{PHYS380-2011}}
#Analytically calculate the entropy of a Gaussian random variable.
#We start with a simple problem. In class, we have defined the mutual information between <math>X</math> and <math>Y</math> as a difference between a marginal and a conditional entropy, <math>I[X;Y]=S[X]- S[X|Y]</math>. Rewrite this expression to depend only on unconditional entropies. What does it say about the relation between the joint entropy of two variables and the two marginal entropies?
#How much information can a spiking neuron transmit? This is limited from above by its entropy rate. Let's represent a neuron as releasing action potentials with a Poisson process with a certain rate <math>r</math>, and let's calculate the entropy rate of the Poisson process. First represent this process by discretizing time in intervals <math>\Delta t</math>. Explain why the entropy of the Poisson generated sequence of duration <math>T</math> (or, alternatively, <math>n=T/\Delta t</math> symbols) is exactly proportional to time, that is <math>S=sn</math>, where <math>s</math> is some constant. Thus we only need to calculate the entropy of a single symbol, this <math>s</math>, in order to find the entropy rate as <math>R=\frac{sT}{\Delta t T}</math>. Does this rate have a finite value as <math>\Delta t\to0</math>? Why or why not?
#''Graduate students:'' Suppose now the neuron has what's called a ''refractory period''. That is, after a spike, a neuron cannot fire for the time <math>\tau_r</math>. What is the entropy rate of such neuron?
#Consider information transmission in a simple model of a molecular circuit. The input signal of <math>s</math> molecules is distributed as a Gaussian random variable with the mean <math>\mu_s</math> and the variance <math>\sigma_s</math>. The elicited mean response is <math>\mu_r=g\mu_s</math>, and it has the usual counting (a.k.a, Poisson or square root) fluctuations around the mean. Assuming that the response molecule count is large, <math>r\gg1</math>, what is the mutual information between <math>s</math> and <math>r</math>? (To answer this, you will need to understand what the marginal distribution of <math>r</math> is.) Now suppose that there are <math>N</math> independent signals <math>s_n</math>, and each is measured by an independent response <math>r_n</math>, such that, as above, <math>\mu_{r_n}=g_n\mu_{s_n}</math>. What is the total mutual information between all signals on the one hand, and all responses on the other?

← Older revision	Revision as of 16:28, 4 July 2018
(No difference)

Physics 380, 2011: Homework 6 - Revision history

Ilya: 1 revision imported

nemenman>Ilya at 02:07, 3 October 2011