Nemenman et al., 2002

From Ilya Nemenman: Theoretical Biophysics @ Emory
Jump to: navigation, search

Back to the full Publications list.

I Nemenman, F Shafee, and W Bialek. "Entropy and inference, revisited." In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Adv. Neural Inf. Proc. Syst. 14, Cambridge, MA, 2002. MIT Press. PDF, arXiv.

Abstract
We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.