Entropy estimation methods
Revision as of 16:54, 14 January 2012 by nemenman>Ilya
This is an incomplete list of references on methodology of entropy estimation using coincidence-counting methods, similar to our NSB method.
The original Ma coincidence-counting method for entropy estimation in the microcanonical ensemble
- S Ma. Calculation of entropy from data of motion. J. Stat. Phys., 26:221-240, 1981.
NSB coincidence-counting method
- Original introduction of the method
- I Nemenman, F Shafee, and W Bialek. "Entropy and inference, revisited." In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Adv. Neural Inf. Proc. Syst. 14, Cambridge, MA, 2002. MIT Press. Abstract.
- Asymptotic analysis of the method
- I Nemenman. Inference of entropies of discrete random variables with unknown cardinalities. IEEE Trans. Inf. Thy., submitted 2005. Abstract.
- Proof of concept application to neural data
- I Nemenman, W Bialek, and R de Ruyter van Steveninck. Entropy and information in neural spike trains: Progress on the sampling problem. Phys. Rev. E, 69:056111, 2004. Abstract.
- Reshuffling technique to further decrease the bias
- While the reshuffling procedure seems to work well for LGN and simulated Poisson data, for which Panzeri et al. have tested it, I am a bit skeptical that the procedure should be recommended as a general tool. It relies on the assumption that the bias in true and reshuffled data is the same, which may or may not be true. I think it will not be true, for example, for structured, refractory data with intricate temporal correlations.
- M Montemurro, R Senatore, and S Panzeri. Tight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques. Neural Comp. 19:2913–2957, 2007. PDF.
- S Panzeri, R Senatore, M Montemurro, and R Petersen. Correcting for the Sampling Bias Problem in Spike Train Information Measures. J Neurophysiol 98: 1064–1072, 2007. PDF.
