Cramér-Rao Bound: Information Inequality, Achievability, and Sharper Variants

For any unbiased estimator $\hat\theta$ of $\theta$,

$\mathrm{Var}_\theta(\hat\theta) \geq \frac{1}{n I(\theta)}.$

Equality at a fixed $\theta$ holds if and only if there exists a function $a(\theta) \neq 0$ such that

$\sum_{i=1}^n s(X_i; \theta) = a(\theta) (\hat\theta - \theta) \quad \mathbb P_\theta\text{-a.s.}$

Equality at every $\theta$ in an open neighborhood characterizes one-parameter exponential families.

The score is the direction in which the log-likelihood rises fastest. An unbiased estimator must be sensitive to the parameter in the same direction (since perturbing $\theta$ shifts $\mathbb E[\hat\theta]$ by exactly the same amount). Cauchy-Schwarz says this collinearity has a cost: the unbiased estimator's variance must absorb at least the inverse of the score's variance. High Fisher information means the score is large in magnitude and tightly concentrated, so the data is informative and small variance is achievable.

This is the central inequality of classical estimation theory. Combined with the asymptotic efficiency of the MLE (see asymptotic statistics), it identifies $1/(nI(\theta))$ as the universal asymptotic floor for unbiased estimation under regularity. Every alternative estimator has to be benchmarked against it: if $\mathrm{Var}(\hat\theta_{\mathrm{alt}}) > 1/(nI(\theta))$ asymptotically, the MLE wins; if it equals it, both are asymptotically efficient. The bound also identifies the parameters that are hard to estimate (small $I(\theta)$) and the ones that are easy (large $I(\theta)$).

Biased estimators escape the bound. The James-Stein estimator beats the sample mean in MSE for $d \geq 3$; ridge regression, LASSO, and posterior mean estimators all introduce bias in exchange for variance reduction. The Cramér-Rao bound governs only the unbiased class.

Non-regular families. For $X \sim U(0, \theta)$ the support depends on $\theta$, $\partial_\theta \log p$ does not exist almost everywhere, and the bound does not apply. The MLE $\hat\theta = X_{(n)}$ has variance $\theta^2 / [n(n+2)] \cdot (1 + o(1))$, which is $\Theta(\theta^2 / n^2)$, faster than the $1/n$ parametric rate. The bound is silent here; use the Hammersley-Chapman-Robbins bound instead.

Bound not attained at finite $n$. Outside one-parameter exponential families, no unbiased estimator attains the bound at every $n$. For example, estimating $\sigma$ from $X_1, \ldots, X_n \sim \mathcal N(0, \sigma^2)$: the unbiased estimator $\hat\sigma = c_n \sqrt{\sum X_i^2}$ has variance strictly larger than $1/(nI(\sigma)) = \sigma^2/(2n)$ at finite $n$, with the gap shrinking at rate $1/n$.

For any unbiased estimator $\hat\theta$ of $\theta \in \mathbb R^d$,

$\mathrm{Cov}_\theta(\hat\theta) \succeq \frac{1}{n} I(\theta)^{-1}$

in the Loewner (positive semidefinite) ordering: $\mathrm{Cov}(\hat\theta) - \frac{1}{n} I(\theta)^{-1}$ is positive semidefinite. Equivalently, for every $v \in \mathbb R^d$,

$\mathrm{Var}_\theta(v^\top \hat\theta) \geq \frac{1}{n} v^\top I(\theta)^{-1} v.$

The matrix bound is the simultaneous lower bound on the variance of every linear combination of the parameter estimates. Parameters that are jointly poorly identified (small eigenvalues of $I(\theta)$ in their direction) have correspondingly large minimum-variance bounds. The inverse Fisher matrix $I(\theta)^{-1}$ is the covariance template that the optimal unbiased estimator must equal or exceed in every direction.

Real models almost always have multiple parameters. The matrix bound governs how trade-offs between parameters propagate. A model whose Fisher matrix has a small eigenvalue is weakly identified in the corresponding direction: any unbiased estimator must accept large variance in that direction. This connects directly to ill-conditioned regression and to the natural gradient (which preconditions descent directions by $I^{-1}$ to remove this distortion).

The bound applies to unbiased estimators only. The multivariate biased-estimator version (next theorem) replaces $I^{-1}$ with $\mathcal J I^{-1} \mathcal J^\top$ where $\mathcal J = \nabla_\theta \mathbb E[\hat\theta]$ is the Jacobian of the bias-corrected mean. When $I(\theta)$ is singular (rank-deficient, i.e. the model is non-identifiable in some direction), the bound is vacuous in that direction; use a generalized inverse $I^+$ and restrict to identifiable contrasts.

For any (possibly biased) estimator $T(X)$ with mean $\psi(\theta) = \mathbb E_\theta[T(X)]$,

$\mathrm{Cov}_\theta(T) \succeq \frac{1}{n} \mathcal J(\theta) I(\theta)^{-1} \mathcal J(\theta)^\top$

where $\mathcal J(\theta) = \nabla_\theta \psi(\theta)$ is the Jacobian of $\psi$. In the scalar case this reduces to

$\mathrm{Var}_\theta(T) \geq \frac{[\psi'(\theta)]^2}{n I(\theta)}.$

A biased estimator's mean shifts at rate $\psi'(\theta)$ as $\theta$ changes (not at rate $1$ as for unbiased estimators). The Cauchy-Schwarz argument now gives a covariance of $\psi'(\theta)$ between $T$ and the score, so the squared-covariance side of Cauchy-Schwarz becomes $[\psi'(\theta)]^2$ instead of $1$. The bound scales accordingly.

This is the bound that practitioners actually need. Most useful estimators are biased: ridge, LASSO, posterior means, shrinkage, kernel-smoothed estimates. The chain-rule form gives the variance lower bound for any such estimator, parameterized by how its mean depends on $\theta$. It also justifies the delta method: an estimator of $g(\theta)$ has variance lower-bounded by $[g'(\theta)]^2 / (n I(\theta))$, which matches the asymptotic variance of $g(\hat\theta_{\mathrm{MLE}})$ exactly under standard regularity.

The chain-rule bound does not directly bound MSE, only variance. To bound MSE, add the squared bias: $\mathrm{MSE}(T) = \mathrm{Var}(T) + [\psi(\theta) - \theta]^2 \geq [\psi'(\theta)]^2 / (n I(\theta)) + [\psi(\theta) - \theta]^2$. The James-Stein estimator beats the sample mean precisely because its bias is small while its variance reduction is large; the Cramér-Rao bound in either form does not preclude this.

Let $S^{(j)}(\theta) = \partial^j \log p(X \mid \theta) / \partial \theta^j$ for $j = 1, \ldots, k$. Define the Gram matrix $\mathcal B(\theta)$ with entries $\mathcal B_{ij} = \mathbb E[S^{(i)} S^{(j)}]$ (the Bhattacharyya information matrix). Then for any unbiased $\hat\theta$,

$\mathrm{Var}_\theta(\hat\theta) \geq \frac{1}{n} \mathbf 1^\top \mathcal B(\theta)^{-1} \mathbf 1$

where $\mathbf 1 = (1, 0, \ldots, 0)^\top \in \mathbb R^k$ (since only the first-order moment of $\hat\theta$ depends on $\theta$). The $k=1$ case recovers the standard Cramér-Rao bound; $k \geq 2$ gives strictly tighter bounds when the higher-order information matrix is non-degenerate.

The standard Cramér-Rao bound projects $\hat\theta$ onto the first score $S^{(1)}$ alone. The Bhattacharyya bound projects onto the larger linear span $\{S^{(1)}, S^{(2)}, \ldots, S^{(k)}\}$ of higher-order log-likelihood derivatives, capturing more of the unbiased estimator's structure. The bound is sharper because the projection onto a larger subspace has equal-or-larger norm.

Bhattacharyya bounds quantify how much sharpness is left on the table by the standard Cramér-Rao bound. In some non-exponential models, the gap is large at finite $n$. The bound also generalizes naturally to the multivariate case (replace $\mathbf 1$ by an identity-block selection matrix).

For one-parameter exponential families, all higher-order Bhattacharyya bounds collapse to the Cramér-Rao bound (the higher-order derivatives are linearly dependent on the first), so no sharpening is possible. Bhattacharyya helps in non-exponential families where the bound is not attained by the Cramér-Rao argument.

For any unbiased estimator $\hat\theta$ of $\theta$ and any $\delta \neq 0$ such that $p(x \mid \theta + \delta)$ and $p(x \mid \theta)$ are mutually absolutely continuous,

$\mathrm{Var}_\theta(\hat\theta) \geq \sup_{\delta \neq 0} \frac{\delta^2}{\chi^2(p_{\theta + \delta} \,\|\, p_\theta)}$

where $\chi^2(q \,\|\, p) = \mathbb E_p[(q/p - 1)^2] = \int (q - p)^2 / p \, dx$ is the chi-squared divergence. As $\delta \to 0$ in a Cramér-regular family, the bound recovers the Cramér-Rao bound (since $\chi^2(p_{\theta + \delta} \,\|\, p_\theta) = \delta^2 I(\theta) + O(\delta^4)$).

Replace the score (a derivative) with a finite difference. The bound asks: how much can the parameter shift by $\delta$ before the data distribution becomes too distinguishable? An unbiased estimator that tracks $\theta$ exactly must also track shifts $\theta \mapsto \theta + \delta$, so its variance must be at least $\delta^2$ times a precision factor measured by chi-squared distance. The supremum over $\delta$ tightens the bound; for small $\delta$ the bound recovers the local Cramér-Rao bound, for large $\delta$ it can be tighter or sharper at finite $n$.

This is the bound to use when Cramér regularity fails. For $X \sim U(0, \theta)$, the chi-squared divergence between $U(0, \theta)$ and $U(0, \theta + \delta)$ is finite for $\delta < 0$ (and infinite for $\delta > 0$), and computing the supremum gives a non-trivial lower bound on the variance of any unbiased estimator. The HCR bound also handles discrete parameters, irregular boundaries, and any setting where the score is undefined or unbounded.

The chi-squared divergence can be infinite (e.g. when supports are disjoint), in which case the bound for that $\delta$ is zero (vacuous). Choose $\delta$ small enough that $\chi^2$ is finite. For complicated parametric families, the HCR bound is hard to compute analytically; the Cramér-Rao bound is preferred when regularity holds.

For any estimator $\hat\theta(X)$ of $\theta$ (no unbiasedness required),

$\mathbb E_\pi \mathbb E_\theta[(\hat\theta - \theta)^2] \geq \frac{1}{\mathbb E_\pi[n I(\theta)] + I(\pi)}$

where $I(\pi) = \int (\pi'(\theta) / \pi(\theta))^2 \pi(\theta) \, d\theta$ is the Fisher information of the prior with respect to the location family.

The Bayesian Cramér-Rao bound combines two information sources: the data information $\mathbb E_\pi[n I(\theta)]$ (averaged over the prior) and the prior information $I(\pi)$. As the prior becomes flat (informationless), $I(\pi) \to 0$ and the bound recovers a Bayes-averaged Cramér-Rao bound. As the data dominates, the prior information becomes negligible and the asymptotic Bayes risk matches the asymptotic frequentist Cramér-Rao bound.

The van Trees inequality is the standard tool for proving minimax lower bounds. Combined with a careful choice of prior (often supported on a small ball around the parameter of interest), it yields explicit lower bounds on the worst-case Bayes risk, which dualizes to a lower bound on the minimax risk. This is one of the two main routes to minimax lower bounds, the other being Le Cam's two-point method.

The bound requires the prior to vanish at the boundary (so the integration-by-parts has no boundary term). Improper priors (uniform on the whole real line) violate this; a careful localization argument is needed. For multidimensional $\theta$, the matrix version replaces sums with the matrix sum $\mathbb E_\pi[n I(\theta)] + I(\pi)$ in the appropriate Loewner sense.