Percy Liang
Michael I. Jordan
ABSTRACT
Statistical and computational concerns have motivated parameter estimators based on various forms of likelihood, e.g., joint, conditional, and pseudolikelihood. In this paper, we present a unified framework for studying these estimators, which allows us to compare their relative (statistical) efficiencies. Our asymptotic analysis suggests that modeling more of the data tends to reduce variance, but at the cost of being more sensitive to model misspecification. We present experiments validating our analysis.
- Besag, J. (1975). The analysis of non-lattice data. The Statistician, 24, 179--195.Google Scholar
Cross Ref
- Bouchard, G., & Triggs, B. (2004). The trade-off between generative and discriminative classifiers. International Conference on Computational Statistics (pp. 721--728).Google Scholar
- Lafferty, J., McCallum, A., & Pereira, F. (2001). Conditional random fields: Probabilistic models for segmenting and labeling data. International Conference on Machine Learning (ICML). Google ScholarDigital Library
- Lasserre, J. A., Bishop, C. M., & Minka, T. P. (2006). Principled hybrids of generative and discriminative models. Computer Vision and Pattern Recognition (CVPR) (pp. 87--94). Google ScholarDigital Library
- Liang, P., Klein, D., & Jordan, M. I. (2008). Agreement-based learning. Advances in Neural Information Processing Systems (NIPS).Google Scholar
- Lindsay, B. (1988). Composite likelihood methods. Contemporary Mathematics, 80, 221--239.Google ScholarCross Ref
- McCallum, A., Pal, C., Druck, G., & Wang, X. (2006). Multi-conditional learning: Generative/discriminative training for clustering and classification. Association for the Advancement of Artificial Intelligence (AAAI). Google ScholarDigital Library
- Ng, A. Y., & Jordan, M. I. (2002). On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. Advances in Neural Information Processing Systems (NIPS).Google Scholar
- Sutton, C., & McCallum, A. (2005). Piecewise training of undirected models. Uncertainty in Artificial Intelligence (UAI).Google Scholar
- van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.Google Scholar
- Varin, C. (2008). On composite marginal likelihoods. Advances in Statistical Analysis, 92, 1--28.Google ScholarCross Ref
- Wainwright, M. (2006). Estimating the "wrong" graphical model: Benefits in the computation-limited setting. Journal of Machine Learning Research, 7, 1829--1859. Google ScholarDigital Library
- Wainwright, M., Jaakkola, T., & Willsky, A. (2003). Treereweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching. Artificial Intelligence and Statistics (AISTATS).Google Scholar
- Wainwright, M., & Jordan, M. I. (2003). Graphical models, exponential families, and variational inference (Technical Report). Department of Statistics, University of California at Berkeley.Google Scholar
Index Terms
- An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators
Recommendations
Performance analysis of multivariate complex amplitude estimators
Part IIWe consider multivariate complex amplitude estimation in the presence of unknown interference and noise. Two multivariate approaches [Maximum Likelihood (ML) and Capon] are provided. We derive the closed-form expression of the Crame´r-Rao bound (...
Asymptotic approximation of kernel-type estimators with its application
Sufficient conditions are given under which a generalized class of kernel-type estimators allows asymptotic approximation on the modulus of continuity. This generalized class includes sample distribution function, kernel-type estimator of density ...
Asymptotic properties of the Bayes modal estimators of item parameters in item response theory
Asymptotic cumulants of the Bayes modal estimators of item parameters using marginal likelihood in item response theory are derived up to the fourth order with added higher-order asymptotic variances under possible model misspecification. Among them, ...
Comments