https://nemenmanlab.org/~ilya/index.php?action=history&feed=atom&title=Entropy_estimation_methodsEntropy estimation methods - Revision history2026-05-17T09:37:45ZRevision history for this page on the wikiMediaWiki 1.31.0https://nemenmanlab.org/~ilya/index.php?title=Entropy_estimation_methods&diff=787&oldid=prevIlya: 1 revision imported2018-07-04T16:28:45Z<p>1 revision imported</p>
<table class="diff diff-contentalign-left" data-mw="interface">
<tr class="diff-title" lang="en">
<td colspan="1" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="1" style="background-color: #fff; color: #222; text-align: center;">Revision as of 16:28, 4 July 2018</td>
</tr><tr><td colspan="2" class="diff-notice" lang="en"><div class="mw-diff-empty">(No difference)</div>
</td></tr></table>Ilyahttps://nemenmanlab.org/~ilya/index.php?title=Entropy_estimation_methods&diff=786&oldid=prevnemenman>Ilya at 21:54, 14 January 20122012-01-14T21:54:53Z<p></p>
<p><b>New page</b></p><div>This is an incomplete list of references on methodology of entropy estimation using coincidence-counting methods, similar to our NSB method.<br />
<br />
==The original Ma coincidence-counting method for entropy estimation in the microcanonical ensemble==<br />
:S Ma. Calculation of entropy from data of motion. ''J. Stat. Phys.'', '''26''':221-240, 1981.<br />
<br />
==[http://nsb-entropy.sf.net NSB coincidence-counting method]==<br />
:;Original introduction of the method<br />
::I Nemenman, F Shafee, and W Bialek. "Entropy and inference, revisited." In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, ''Adv. Neural Inf. Proc. Syst.'' '''14''', Cambridge, MA, 2002. MIT Press. [[Nemenman et al., 2002 | Abstract]]. <br />
:;Asymptotic analysis of the method<br />
::I Nemenman. Inference of entropies of discrete random variables with unknown cardinalities. IEEE Trans. Inf. Thy., submitted 2005. [[Nemenman-IEEEIT | Abstract]]. <br />
:;Proof of concept application to neural data<br />
::I Nemenman, W Bialek, and R de Ruyter van Steveninck. Entropy and information in neural spike trains: Progress on the sampling problem. ''Phys. Rev. E'', '''69''':056111, 2004. [[Nemenman et al., 2004 | Abstract]]. <br />
:;Reshuffling technique to further decrease the bias<br />
::While the reshuffling procedure seems to work well for LGN and simulated Poisson data, for which Panzeri et al. have tested it, I am a bit skeptical that the procedure should be recommended as a general tool. It relies on the assumption that the bias in true and reshuffled data is the same, which may or may not be true. I think it will not be true, for example, for structured, refractory data with intricate temporal correlations.<br />
::#M Montemurro, R Senatore, and S Panzeri. Tight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques. ''Neural Comp''. '''19''':2913–2957, 2007. [[media:montemurro-etal-07.pdf | PDF]]. <br />
::#S Panzeri, R Senatore, M Montemurro, and R Petersen. Correcting for the Sampling Bias Problem in Spike Train Information Measures. ''J Neurophysiol'' '''98''': 1064–1072, 2007. [[media:Panzeri-etal-07.pdf | PDF]].<br />
:;Application to neural data and introduction of partitioning method<br />
::I Nemenman, GD Lewen, W Bialek, RR de Ruyter van Steveninck. Neural Coding of Natural Stimuli: Information at Sub-Millisecond Resolution. ''PLoS Comput Biol'' '''4'''(3): e1000025, 2008. [[media:Nemenman-etal-08.pdf | PDF]], [http://arxiv.org/abs/q-bio.NC/0612050 arXiv], [[Nemenman et al., 2008 | Abstract]].<br />
<br />
[[Category:Literature review]]</div>nemenman>Ilya