Entropy estimation methods

This is an incomplete list of references on methodology of entropy estimation using coincidence-counting methods, similar to our NSB method.

The original Ma coincidence-counting method for entropy estimation in the microcanonical ensemble

S Ma. Calculation of entropy from data of motion. J. Stat. Phys., 26:221-240, 1981.

NSB coincidence-counting method

Original introduction of the method

I Nemenman, F Shafee, and W Bialek. "Entropy and inference, revisited." In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Adv. Neural Inf. Proc. Syst. 14, Cambridge, MA, 2002. MIT Press. Abstract.

Asymptotic analysis of the method

I Nemenman. Inference of entropies of discrete random variables with unknown cardinalities. IEEE Trans. Inf. Thy., submitted 2005. Abstract.

Proof of concept application to neural data

I Nemenman, W Bialek, and R de Ruyter van Steveninck. Entropy and information in neural spike trains: Progress on the sampling problem. Phys. Rev. E, 69:056111, 2004. Abstract.

Reshuffling technique to further decrease the bias

While the reshuffling procedure seems to work well for LGN and simulated Poisson data, for which Panzeri et al. have tested it, I am a bit skeptical that the procedure should be recommended as a general tool. It relies on the assumption that the bias in true and reshuffled data is the same, which may or may not be true. I think it will not be true, for example, for structured, refractory data with intricate temporal correlations.

1. M Montemurro, R Senatore, and S Panzeri. Tight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques. Neural Comp. 19:2913–2957, 2007. PDF.
2. S Panzeri, R Senatore, M Montemurro, and R Petersen. Correcting for the Sampling Bias Problem in Spike Train Information Measures. J Neurophysiol 98: 1064–1072, 2007. PDF.

Application to neural data and introduction of partitioning method

I Nemenman, GD Lewen, W Bialek, RR de Ruyter van Steveninck. Neural Coding of Natural Stimuli: Information at Sub-Millisecond Resolution. PLoS Comput Biol 4(3): e1000025, 2008. PDF, arXiv, Abstract.