Maximum Likelihood Estimation: Theory, Information Identity, and Asymptotic Efficiency

Under regularity, the score has mean zero and Fisher information equals the negative expected Hessian:

$\mathbb E_\theta[s(\theta; X)] = 0, \qquad \mathrm{Cov}_\theta(s(\theta; X)) = I(\theta) = -\mathbb E_\theta[\nabla^2_\theta \log p(X; \theta)].$

Equivalently, $\mathbb E_\theta[\nabla_\theta s(\theta; X) + s(\theta; X) s(\theta; X)^\top] = 0$, the second Bartlett identity.

Differentiate $1 = \int p(x; \theta) \, d\mu(x)$ once: $0 = \int \nabla_\theta p = \int s \cdot p \, d\mu = \mathbb E[s]$. Differentiate again:

$0 = \int \nabla^2_\theta p \, d\mu = \int (\nabla_\theta s + s s^\top) p \, d\mu = \mathbb E[\nabla_\theta s] + \mathbb E[s s^\top].$

Since $\mathbb E[\nabla_\theta s] = \mathbb E[\nabla^2_\theta \log p]$, this rearranges to $-\mathbb E[\nabla^2 \log p] = \mathbb E[s s^\top] = I(\theta)$. The identity is bookkeeping that turns one expression for $I$ into the other; the content is that you can interchange differentiation and integration.

The information identity is the linchpin of three subsequent results:

1. Asymptotic normality of the MLE uses $-\mathbb E[\nabla^2 \log p] = I(\theta)$ to identify the variance.
2. Cramér-Rao uses $\mathrm{Cov}(s) = I(\theta)$ in the Cauchy-Schwarz step.
3. Wilks' theorem uses both forms when expanding $\log L(\hat\theta) - \log L(\theta_0)$ to second order.

When the identity fails (parameter on a boundary, support depending on $\theta$, non-smooth $\log p$), every classical asymptotic for the MLE fails along with it, and you need M-estimator machinery (sandwich variance, cube-root rates, mixture-$\chi^2$ limits) instead.

The interchange step requires more than two derivatives existing. Standard counterexamples: (1) Uniform$(0, \theta)$ with support $[0, \theta]$ depends on $\theta$, so $\nabla \int p = 0$ but $\int \nabla p \neq 0$; the score is not even well-defined and the MLE is $X_{(n)}$ with rate $n$, not $\sqrt n$. (2) Models with a discontinuity in $p$ at a $\theta$-dependent threshold (change-point models) violate the same interchange; the MLE has a non-Gaussian limit at rate $n$. (3) Mixture models on the boundary ($\pi = 0$) make $I(\theta)$ singular and the second identity meaningless.

Under Wald's conditions, the MLE is consistent:

$\hat\theta_n \xrightarrow{P} \theta_0 \quad \text{as } n \to \infty.$

If $\Theta$ is open (rather than compact), the same conclusion holds when $\ell_n$ is concave and $\theta_0$ is the unique maximizer of $\theta \mapsto \mathbb E_{\theta_0}[\log p(X; \theta)]$.

The empirical log-likelihood per observation $\bar\ell_n(\theta) = n^{-1} \sum_i \log p(X_i; \theta)$ is a sample average of i.i.d. terms. By the law of large numbers, $\bar \ell_n(\theta) \to L(\theta) := \mathbb E_{\theta_0}[\log p(X; \theta)]$ pointwise. The Kullback-Leibler inequality gives

$L(\theta_0) - L(\theta) = D_{\mathrm{KL}}(p_{\theta_0} \| p_\theta) \geq 0,$

with equality only when $p_{\theta_0} = p_\theta$, i.e. (by identifiability) $\theta = \theta_0$. So $\theta_0$ uniquely maximizes the limit objective. Under uniform convergence (Wald's compactness + envelope, or concavity), the maximizer of $\bar\ell_n$ converges to the maximizer of $L$, which is $\theta_0$.

Consistency is the minimum bar for any estimator and the prerequisite for asymptotic normality (which is local around $\theta_0$). The proof structure has three parts: LLN gives pointwise convergence; uniformity comes from compactness or concavity; KL identifies the maximizer. It is the template every modern consistency proof reuses (M-estimators, GMM, EM, deep learning's empirical loss).

(1) Non-identifiability. Mixture models with label-switching: $(\pi, \mu_1, \mu_2)$ and $(1-\pi, \mu_2, \mu_1)$ give the same density. The KL gap collapses on the orbit, and the MLE is not consistent for any single labeling. (2) Boundary blow-up. In Gaussian mixtures, sending one component's variance to zero with its mean equal to a data point makes $L_n \to \infty$. The MLE does not exist; constrained or penalized variants are needed. (3) Neyman-Scott. Parameter dimension growing with $n$ (one mean per pair of observations, one shared variance) gives an inconsistent variance MLE. Profile or modified likelihoods recover consistency. (4) Misspecification. If $p_{\theta_0}$ is not in the model family, the MLE converges to the least-false parameter $\arg\min_\theta D_{\mathrm{KL}}(P_0 \| p_\theta)$, not to any "true" $\theta_0$ (see QMLE below).

Under regularity and consistency, the MLE is asymptotically normal:

$\sqrt{n}(\hat\theta_n - \theta_0) \xrightarrow{d} \mathcal N\!\left(0, I(\theta_0)^{-1}\right).$

Equivalently, $\hat\theta_n \approx \mathcal N(\theta_0, I(\theta_0)^{-1}/n)$ for large $n$. The variance $I(\theta_0)^{-1}/n$ is the Cramér-Rao lower bound, attained by the MLE in the limit.

Taylor-expand the score equation around $\theta_0$:

$0 = \frac{1}{n} \sum_i s(\hat\theta_n; X_i) \approx \frac{1}{n} \sum_i s(\theta_0; X_i) + \left[\frac{1}{n} \sum_i \nabla s(\theta_0; X_i)\right] (\hat\theta_n - \theta_0).$

The first term, scaled by $\sqrt n$, is a CLT for an i.i.d. mean-zero sum with covariance $I(\theta_0)$ (Bartlett identity I), so $n^{-1/2} \sum_i s(\theta_0; X_i) \xrightarrow{d} \mathcal N(0, I(\theta_0))$. The bracket converges in probability to $\mathbb E[\nabla s(\theta_0; X)] = -I(\theta_0)$ (Bartlett identity II + LLN). Solving,

$\sqrt n (\hat\theta_n - \theta_0) \approx I(\theta_0)^{-1} \cdot n^{-1/2} \sum_i s(\theta_0; X_i) \xrightarrow{d} \mathcal N(0, I(\theta_0)^{-1} I(\theta_0) I(\theta_0)^{-1}) = \mathcal N(0, I(\theta_0)^{-1}).$

Asymptotic normality is the engine behind every classical inferential procedure built on the MLE: Wald confidence intervals $\hat\theta_n \pm z_{1-\alpha/2} \sqrt{I(\hat\theta_n)^{-1}/n}$, Wald hypothesis tests, model comparison via AIC and BIC (which use the asymptotic distribution of the maximized log-likelihood). The functional form $\mathcal N(0, I^{-1})$ also plugs straight into the delta method for any smooth $g(\hat\theta_n)$, giving CIs for derived quantities (odds ratios, hazard ratios, predicted probabilities).

Asymptotic normality fails in four canonical cases. (1) $\theta_0$ sits on the boundary of $\Theta$ (variance equal to zero, mixture proportion equal to zero): the limit becomes a projection of a Gaussian onto a tangent cone (Self-Liang 1987). (2) Fisher information is singular (flat likelihood, weak identification): the rate slows below $\sqrt n$ or the limit becomes non-Gaussian. (3) The model is misspecified: replace $I(\theta_0)^{-1}$ with the sandwich $V^{-1} A V^{-\top}$ (QMLE, below). (4) Finite-sample concerns dominate, especially in high dimensions where the $\sqrt n$ regime requires $n \gg d^2$ in many examples.

Let $\Lambda_n = L_n(\hat\theta_n^{(0)}) / L_n(\hat\theta_n)$ be the likelihood ratio comparing the constrained MLE under $H_0$ to the unconstrained MLE. Under the null,

$-2 \log \Lambda_n = 2 \big(\ell_n(\hat\theta_n) - \ell_n(\hat\theta_n^{(0)})\big) \xrightarrow{d} \chi^2_q,$

where $q$ is the codimension of the null (number of restrictions). Under contiguous alternatives $\theta_n = \theta_0 + h/\sqrt n$, the limit is non-central $\chi^2_q(\delta)$ with non-centrality $\delta = h^\top I(\theta_0)_{\mathrm{rest}} h$, where $I(\theta_0)_{\mathrm{rest}}$ is the restricted block of Fisher information.

Taylor-expand the log-likelihood around the unconstrained MLE:

$\ell_n(\hat\theta_n) - \ell_n(\hat\theta_n^{(0)}) \approx \tfrac 12 (\hat\theta_n - \hat\theta_n^{(0)})^\top n I(\theta_0) (\hat\theta_n - \hat\theta_n^{(0)}) + o_P(1).$

The vector $\sqrt n (\hat\theta_n - \hat\theta_n^{(0)})$ is asymptotically Gaussian, projected onto the orthogonal complement of the null tangent space (a $q$-dimensional subspace). Quadratic forms of $q$-dim Gaussians with the right covariance are exactly $\chi^2_q$.

Wilks' theorem makes the likelihood-ratio test (LRT) practical: you can compute $-2 \log \Lambda_n$ and compare to a $\chi^2_q$ critical value without ever computing Fisher information by hand. It is the basis for nested-model comparison in regression (drop $q$ predictors and the LRT has a $\chi^2_q$ null), in GLMs (deviance differences), in mixed models, in survival analysis (Cox PH), and in modern model selection (BIC penalty derives from the Wilks limit). The non-central $\chi^2$ result under contiguous alternatives gives the asymptotic power of the test; see the LRT vs Wald vs score equivalence in asymptotic-statistics.

Wilks' fails on the same boundaries that asymptotic normality does. Self-Liang (1987) shows that on a parameter-space boundary the limit is a mixture of $\chi^2$ distributions (e.g., for testing $\sigma^2 = 0$ in a mixed model the limit is $\tfrac 12 \chi^2_0 + \tfrac 12 \chi^2_1$). Under non-identifiability (testing whether a mixture component is present, the famous "two-component vs. one-component" test) the limit is non-standard and depends on a nuisance parameter under the alternative (Davies 1977). Don't apply Wilks blindly when the null is on a boundary or weakly identified.

Let $T = T(X_1, \ldots, X_n)$ be unbiased for $\theta \in \mathbb R$. Then

$\mathrm{Var}_\theta(T) \geq \frac{1}{n I(\theta)}.$

In the multivariate case ($\theta \in \mathbb R^d$), $\mathrm{Cov}_\theta(T) \succeq (n I(\theta))^{-1}$ in the Loewner order. An estimator attaining the bound is called efficient. The MLE achieves the bound asymptotically (the previous theorem).

Differentiating $\mathbb E_\theta[T] = \theta$ in $\theta$ gives $\mathrm{Cov}(T, s_n) = 1$ where $s_n = \sum_i s(\theta; X_i)$ is the total score. Cauchy-Schwarz gives $1 = |\mathrm{Cov}(T, s_n)|^2 \leq \mathrm{Var}(T) \cdot \mathrm{Var}(s_n) = \mathrm{Var}(T) \cdot n I(\theta)$. The information sets the price tag on precision: each unit of Fisher information buys a $1/I(\theta)$ unit of variance reduction.

Cramér-Rao is the universal benchmark for unbiased estimators. The MLE attains it asymptotically (third theorem of this section), making MLE the asymptotically minimum-variance unbiased estimator in regular models. It also exposes a deep duality: minimum variance is the inverse of Fisher information, and Fisher information is the local Riemannian metric on the parameter manifold (Rao 1945; see information geometry). The bound's existence is what makes "no unbiased estimator can outperform the MLE" a precise statement rather than folklore.

Cramér-Rao binds unbiased estimators only. Biased estimators can have strictly lower mean-squared error: $\mathrm{MSE} = \mathrm{Var} + \mathrm{Bias}^2$, and a small bias can buy a large variance reduction. The James-Stein estimator is the canonical example (see Confusion below). Modern regularization (ridge, LASSO, weight decay, early stopping) trades a controlled bias for a large variance reduction and routinely beats the MLE in mean-squared error. In high dimensions the MLE is inadmissible and shrinkage is the rule, not the exception.

Suppose $P_0 \neq p_\theta$ for any $\theta$ (the model is misspecified). Define

$V(\theta^*) = -\mathbb E_{P_0}[\nabla^2_\theta \log p_{\theta^*}(X)], \qquad A(\theta^*) = \mathbb E_{P_0}[s(\theta^*; X) \, s(\theta^*; X)^\top].$

Then the MLE converges to the least-false parameter $\theta^*$ and is asymptotically normal with the sandwich variance:

$\sqrt n (\hat\theta_n - \theta^*) \xrightarrow{d} \mathcal N(0, V(\theta^*)^{-1} A(\theta^*) V(\theta^*)^{-\top}).$

Under correct specification, the second Bartlett identity gives $V = A = I(\theta_0)$ and the sandwich collapses to $I(\theta_0)^{-1}$.

Without correct specification, the score still has mean zero at the least-false parameter (because $\theta^*$ minimizes KL, hence sets the gradient of $\mathbb E_{P_0}[\log p_\theta]$ to zero), but the score covariance and the Hessian are no longer linked by the information identity. The Taylor argument that gave $I^{-1}$ in the well-specified case now gives the more general M-estimator sandwich $V^{-1} A V^{-\top}$. Reporting $I^{-1}$ (or, equivalently, $J_n^{-1}$) when the model is wrong gives anti-conservative confidence intervals that miss their nominal coverage.

Real models are always at least slightly wrong. Heteroskedastic linear regression fit by OLS, logistic regression with the wrong link, mixture models with the wrong number of components, neural networks fit to data not generated by them: all are quasi-MLE. The sandwich variance gives the correct asymptotic covariance, and robust standard errors (Huber-White) are the implementation of this in econometric practice. Reporting $I^{-1}$-based standard errors when the model is misspecified is the most common subtle error in applied likelihood inference.

Even QMLE requires identifiability of the least-false parameter $\theta^*$ and finite second moments of the score under $P_0$. Without uniqueness of $\theta^*$ (e.g., the KL surface has multiple equal-depth minima) the MLE has no well-defined limit. Without finite moments (heavy-tailed data and a Gaussian model) the sandwich blows up. Wilks' theorem also fails under misspecification: $-2 \log \Lambda_n$ converges to a weighted sum of $\chi^2_1$ (Vuong 1989) rather than a clean $\chi^2_q$, so naive LRTs over-reject.