A comparative study of parameter estimation methods for statistical natural language processing

This paper presents a comparative study of five parameter estimation algorithms on four NLP tasks. Three of the five algorithms are well-known in the computational linguistics community: Maximum Entropy (ME) estimation with L ₂ regularization, the Averaged Perceptron (AP), and Boosting. We also investigate ME estimation with L ₁ regularization using a novel optimization algorithm, and BLasso, which is a version of Boosting with Lasso (L ₁) regularization. We first investigate all of our estimators on two re-ranking tasks: a parse selection task and a language model (LM) adaptation task. Then we apply the best of these estimators to two additional tasks involving conditional sequence models: a Conditional Markov Model (CMM) for part of speech tagging and a Conditional Random Field (CRF) for Chinese word segmentation. Our experiments show that across tasks, three of the estimators - ME estimation with L ₁ or L ₂ regularization, and AP - are in a near statistical tie for first place.