Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) chooses the parameter estimate ${\hat{\theta }}_{MLE}=\arg {\max }_{\theta }L(\theta \mid x)$ that maximizes the likelihood function, equivalently, the value that maximizes the log-likelihood, ${\hat{\theta }}_{MLE}=\arg {\max }_{\theta }\ell (\theta )$. Under standard regularity conditions, MLE produces estimators that are consistent, asymptotically normal, and asymptotically efficient (attaining the Cramer-Rao lower bound in the large-sample limit), which is why MLE serves as the default estimation method for an enormous range of parametric statistical models, from simple distribution fitting to generalized linear models to complex latent-variable models fit via the EM algorithm.

• MLE's excellent asymptotic properties, consistency, asymptotic normality, asymptotic efficiency, are guaranteed only under standard regularity conditions (e.g. the parameter isn't at a boundary, the model is correctly specified) -- when those conditions fail, MLE can behave poorly, and specialized corrections or alternative estimators may be needed.
• For many models, the maximum likelihood estimate has no closed-form algebraic solution and must be found numerically, using iterative optimization methods like Newton-Raphson, Fisher scoring, or the EM algorithm, particularly for models involving latent variables or missing data.
• Maximum likelihood is a fundamentally different philosophy from method-of-moments estimation, MLE uses the full distributional assumption to extract as much information as possible from the data, generally making it more efficient than method of moments when the assumed model is correct, but also more sensitive to model misspecification.