Physics 380, 2011: Homework 6

Back to the main Teaching page.

Back to Physics 380, 2011: Information Processing in Biology.

1. Analytically calculate the entropy of a Gaussian random variable.
2. We start with a simple problem. In class, we have defined the mutual information between $X$ and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle Y} as a difference between a marginal and a conditional entropy, Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle I[X;Y]=S[X]- S[X|Y]} . Rewrite this expression to depend only on unconditional entropies. What does it say about the relation between the joint entropy of two variables and the two marginal entropies?
3. How much information can a spiking neuron transmit? This is limited from above by its entropy rate. Let's represent a neuron as releasing action potentials with a Poisson process with a certain rate Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle r} , and let's calculate the entropy rate of the Poisson process. First represent this process by discretizing time in intervals Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta t} . Explain why the entropy of the Poisson generated sequence of duration Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle T} (or, alternatively, Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle n=T/\Delta t} symbols) is exactly proportional to time, that is Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle S=sn} , where $s$ is some constant. Thus we only need to calculate the entropy of a single symbol, this Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s} , in order to find the entropy rate as $R={\frac {sT}{\Delta tT}}$. Does this rate have a finite value as $\Delta t\to 0$? Why or why not?
4. Graduate students: Suppose now the neuron has what's called a refractory period. That is, after a spike, a neuron cannot fire for the time $\tau _{r}$. What is the entropy rate of such neuron?
5. Consider information transmission in a simple model of a molecular circuit. The input signal of $s$ molecules is distributed as a Gaussian random variable with the mean $\mu _{s}$ and the variance $\sigma _{s}$. The elicited mean response is $\mu _{r}=g\mu _{s}$, and it has the usual counting (a.k.a, Poisson or square root) fluctuations around the mean. Assuming that the response molecule count is large, $r\gg 1$, what is the mutual information between $s$ and $r$? (To answer this, you will need to understand what the marginal distribution of $r$ is.) Now suppose that there are $N$ independent signals $s_{n}$, and each is measured by an independent response $r_{n}$, such that, as above, $\mu _{r_{n}}=g_{n}\mu _{s_{n}}$. What is the total mutual information between all signals on the one hand, and all responses on the other?