Ilya: 1 revision imported

2018-07-04T16:28:41Z

1 revision imported

nemenman>Ilya at 02:04, 4 October 2012

2012-10-04T02:04:21Z

New page

{{PHYS380-2011}}

In these lectures, we cover some background on information theory. A good physics style introduction to this problem can be found in the upcoming book by Bialek (Bialek 2010). A very nice, and probably still the best, introduction to information theory as a theory of communication is (Shannon and Weaver, 1949). A standard and very good textbook on information theory is (Cover and Thomas, 2006).

==Finishing the previous leture==
*Mutual information: what if we want to know about a variable <math>x</math>, but instead are measuring a variable <math>y</math>. How much are we learning about <math>x</math> then? This is given by the difference of entropies of <math>x</math> before and after the measurement: <math>\begin{array}{ll}I[X;Y]&=S[X]-\langle S[X|Y]\rangle_y\\&=S[X]+S[Y]-S[X,Y]\\&=\langle\log_2\frac{P(x,y)}{P(x)P(y)}\end{array}</math>.
*Meaning of mutual information: mutual information of 1 bit between two variables means that by querying one of them as much as possible, we can get one bit of information about the other.
*Properties of mutual information
*#Limits: <math>0\le I[X;Y]\le \min(S[X],S[X])</math>. Note that the first inequality becomes an equality iff the two variables are completely statistically independent.
*#Mutual information is well-defined for continuous variables.
*#Reparameterization invariance: for any <math>\xi=\xi(x),\, \eta=\eta(y)</math>, the following is true <math>I[X;Y]=I[\Xi;\Eta]</math>.
*#Data processing inequality: For <math>P(x,y,z)=P(x)P(y|x)P(z|y)</math>, <math>I[X;Z]\le \min (I[X;Y], I[Y;Z])</math>. That is, information cannot get created in a transformation of a variable, whether deterministic or probabilistic.
*#Information rate: Information is also an extensive quantity, so that it makes sense to define an information rate <math>I_0=\lim_{n\to\infty}I[X_1,\dots,X_n;Y_1\dots Y_n]/n</math>.
*Mutual information of a bivariate normal with a correlation coefficient <math>\rho</math> is <math>I=-1/2 \log_2(1-\rho^2)</math>.
*For Gaussian variables <math>y=g(x+\eta)</math>, where <math>x</math> is the signal, <math>y</math> is the response, and <math>\eta</math> is the noise related to the input, <math>I[X;Y]=\frac{1}{2}\log_2\left(1+\frac{\sigma^2_x}{\sigma^2_\eta}\right)=\frac{1}{2}\log_2(1+SNR)</math>.

==Warmup question==
#For transmitting information through a synthetic transcriptional circuit in ''E. coli'' (Guet et al., 2002) -- see picture on the board -- which of the following quantities might constrain the mutual information between the chemical signal and the expressed reporter response?
#*The mean molecular copy number of the reporter molecule.
#*The mean molecular copy number of the other, non-reporter genes.
#*The probability distribution of the input signals.

In this lecture, we will try to derive the limits on the quality of information processing in molecular circuits.

==Main Lecture==
*We follow the discussion of Ziv et al., 2007.
*Consider a chain: signal s -> mean response <math>\rho</math> -> actual noisy response r. Due to signal processing inequality, <math>I[S,R]\le I[\rho, R]</math>.
*Assuming that <math>\rho\gg 1</math>, and following the general formula for noise propagation from two lectures ago, we get <math>r=\rho+\eta</math>, where <math>\sigma^2_\eta=\rho</math>.
*Simple counting argument suggests that for N molecules with <math>\sqrt{N}</math> error, there are <math>\sqrt{N}</math> distinguishable states, and the information is limited by <math>I\approx 1/2 \log_2 N</math>.
*For a fixed mean response N, we can calculate <math>P(\rho)</math> that maximizes <math>I[\rho;R]</math>. We get <math>P(\rho)=\frac{1}{(2\pi \rho N)^{1/2}}\exp[-\rho/(2N)]</math>. And the information becomes <math>I\approx 1/2 \log_2 N+O(1/N)</math>.
*This is the best possible result. But in a biochemical system not every <math>P(\rho)</math> is possible. Biochemical systems are modeled typically with Hill activation/suppression dynamics <math>\frac{dx}{dt}=\frac{Vy^n}{y_0^n+y^n}</math> (activation of x by y) or <math>\frac{dx}{dt}=\frac{V}{y_0^n+y^n}</math> (suppression of x by y). And different values of the chemical signals may switch the binding/MIchaelis constant <math>y_0</math>. To achieve good information transmission, one would need to make sure that mean responses to different chemical inputs are well-separated and narrow -- but one wouldn't be able to have the optimal distribution of <math>g</math> as stated above (see picture on board how these distribution looks).
*Ziv et al., 2007, have shown that the inability of biochemical networks to have the optimal <math>P(g)</math> is not very important. Even with this constraint, the maximum information that can be transmitted through these systems is still close to <math>\min(S[S], 1/2\log_2N)</math>.
*Biochemical systems can be very efficient in transmitting information, with their intrinsic stochastic noisiness being, essentially, the main constraint.

← Older revision	Revision as of 16:28, 4 July 2018
(No difference)

Physics 380, 2011: Lecture 10 - Revision history

Ilya: 1 revision imported

nemenman>Ilya at 02:04, 4 October 2012