Ilya: 1 revision imported

2018-07-04T16:28:42Z

1 revision imported

nemenman>Ilya: /* Main lecture */

2011-11-17T14:52:27Z

‎Main lecture

New page

{{PHYS380-2011}}
This is the first lecture in the "adaptation" block.

==Warmup question==

==Main lecture==
*Consider again a system where response is driven by a nonlinear function of the input, with noise. For example <math>\frac{dr}{dt}=f(s)-kr+\eta</math>.
*In the steady state, <math>\bar{r}=\frac{f(s)}{k}</math>. Is there a way to choose the function <math>f(s)</math> in an optimal way, so that the information between <math>s,r</math> is maximized?
*We have to specify what we mean by maximizing information, since, for any ivertible <math>f</math>, the information between the entire time series, <math>I[\{s(t)\},\{r(t)\}]={\rm const}</math>, independent of <math>f</math>.
*Let's look to maximize the same-time information, <math>I[s(t),r(t)]={\rm const}</math>. We assume that <math>r\to 0</math>, so that the system operates, basically, at quasi-steady-state. Then <math>I[s,r]= S[r]-S[r|s]</math>.
*The second term is given by the entropy of the gaussian noise <math>\eta</math>, <math>S[s|r]=\frac{1}{2}\log_22\pi e\sigma_\eta^2</math>.
*The first term is <math>S[r] =- \int dr P(r)\log_2P(r)</math>. But <math>P(r)= \int ds\, P (r|s)P(s)</math>. We can assume that the noise is small, so that </math>P(r|s)</math> is much narrower than <math>P(s)</math>. Then <math>P(r)\approx P(\bar{r})</math>
*Then maximization of MI means maximization of <math>S[\bar{r}]</math>. That is, <math>\bar{r}</math> must be uniformly distributed, and so <math>f(s)=C(s)</math>, where <math>C(s)</math> is the cumulative distribution -- this derivation was done by Laughlin in 1981. SImilar arguments can be made for evolutionary adaptation.
*This is an example of matching. Specifically, one needs to match mid-point and the width of the response curves to the mean and the standard deviation of the signal.
*In more complex cases, when noise is not small, or when it is signal-dependent, similar derivations are often possible, but they are not analytic.
*Why do we focus at all on same-time information? What is the general value of the observed sensory information? We will study mechanisms of such matching in a future lecture.
*Actions take time, and so to be useable, information about the outside world must be relevant to what the world will be in the future. That is, our past signals drive past responses, resulting in potentially high <math>I[s_p;r_p]</math>. But instead we need high <math>I[s_f;r_p]</math> for these responses to be actionable on.
*When signals and responses have short-time correlations only, <math>I[s_f;r_p]\approx I[s(t);r(t)]</math>
*This adds the other adaptation that we already discussed -- filtering with the appropriate time scale.

← Older revision	Revision as of 16:28, 4 July 2018
(No difference)

Physics 380, 2011: Lecture 23 - Revision history

Ilya: 1 revision imported

nemenman>Ilya: /* Main lecture */