Functional Analysis Core

Let $M$ be a subspace of a normed space $X$ and $f: M \to \mathbb{R}$ a bounded linear functional with $\|f\|_M = c$. Then there exists an extension $F: X \to \mathbb{R}$ such that $F|_M = f$ and $\|F\|_X = c$.

In other words, bounded linear functionals on subspaces can always be extended to the whole space without increasing their norm.

You can always "fill in" a linear functional defined on a subspace to the whole space. In finite dimensions this is trivial (extend a basis). In infinite dimensions it is not obvious and requires Zorn's lemma. The norm-preservation is the key: you do not lose any control during the extension.

Hahn-Banach guarantees that the dual space $X^*$ is rich enough to separate points: for any $x \neq y$ in $X$, there exists $f \in X^*$ with $f(x) \neq f(y)$. This is the foundation of duality theory. In ML, it underpins the representer theorem in RKHS: the optimal function in a regularized problem can be expressed in terms of kernel evaluations because evaluation functionals are bounded (and hence extendable).

Hahn-Banach is an existence theorem (via Zorn's lemma). It does not give a constructive recipe for the extension. In non-separable spaces, the extension may not be unique.

Let $\{T_\alpha\}_{\alpha \in A}$ be a family of bounded linear operators from a Banach space $X$ to a normed space $Y$. If the family is pointwise bounded:

$\sup_{\alpha \in A} \|T_\alpha x\| < \infty \quad \text{for every } x \in X$

then it is uniformly bounded:

$\sup_{\alpha \in A} \|T_\alpha\| < \infty$

If a collection of operators does not blow up at any single point, then it cannot blow up anywhere. The completeness of $X$ (Banach space) is essential: it ensures that the "bad points" where operators are large cannot be everywhere dense. This is a Baire category argument: the complement of the set where operators are uniformly bounded is meager.

This theorem is the reason that pointwise convergence of operators implies bounded norms, which is essential for proving convergence of iterative algorithms in function spaces. In approximation theory, it explains why polynomial interpolation can diverge (Faber's theorem): the interpolation operators have unbounded norms, so by Banach-Steinhaus, there must exist continuous functions for which interpolation diverges.

Completeness is essential. In incomplete normed spaces, pointwise boundedness does not imply uniform boundedness. The classic counterexample uses a Hamel basis argument on a dense incomplete subspace.

If $T: X \to Y$ is a surjective bounded linear operator between Banach spaces, then $T$ is an open map: it maps open sets to open sets.

An immediate corollary: if $T$ is also injective (hence bijective), then $T^{-1}$ is automatically bounded. That is, a bijective bounded linear operator between Banach spaces always has a bounded inverse.

A surjective bounded operator cannot "squash" open sets to thin sets. If $T$ hits all of $Y$, then it must hit neighborhoods of every point with neighborhoods. The corollary (bounded inverse) is the infinite-dimensional analog of the fact that invertible matrices have bounded inverses, but here it requires completeness of both spaces.

The open mapping theorem guarantees stability of inverse problems: if a bounded linear operator between Banach spaces is invertible, the inverse is automatically continuous. In optimization, this means solution maps of certain well-posed problems are continuous in the data.

Completeness of both $X$ and $Y$ is essential. Without it, a bijective bounded operator can have an unbounded inverse. The theorem also fails for nonlinear maps.

The closed unit ball of the dual space $X^*$:

$B_{X^*} = \{f \in X^* : \|f\| \leq 1\}$

is compact in the weak-* topology.

In infinite dimensions, the closed unit ball of a Banach space is not compact (in the norm topology). Banach-Alaoglu recovers compactness by switching to a weaker topology. The weak-* topology is the coarsest topology making all evaluation maps $f \mapsto f(x)$ continuous. In this topology, bounded sequences of functionals always have convergent subsequences.

Banach-Alaoglu provides the compactness needed for existence arguments in optimization and variational problems. When you minimize a functional over a bounded set in a dual space, Banach-Alaoglu guarantees that minimizing sequences have convergent subsequences. This is the functional-analytic foundation of many existence proofs in learning theory and optimal transport.

Weak-* compactness is weaker than norm compactness. A sequence converging weak-* need not converge in norm. This distinction matters when you need convergence rates, not just existence.

Let $X, Y$ be Banach spaces and $T: X \to Y$ a linear map. If the graph $\Gamma(T) = \{(x, Tx) : x \in X\}$ is closed in $X \times Y$ (with the product norm), then $T$ is bounded.

For a linear map between Banach spaces, boundedness is equivalent to graph closure. This is a corollary of the open mapping theorem applied to the projection $\Gamma(T) \to X$. It is often the cleanest route to proving boundedness of a concrete operator: you only check that $x_n \to x$ and $T x_n \to y$ together imply $y = Tx$, rather than bounding $\|T x\|$ in terms of $\|x\|$ directly.

The closed graph theorem is the standard tool for proving that differential operators, conditional expectation operators, and closures of unbounded symmetric operators are bounded when they have closed graphs. In ML, it is used to show that certain natural maps between function spaces (e.g., kernel evaluation, sampling operators) are automatically continuous.

Completeness of both $X$ and $Y$ is essential. Without it, a linear map can have a closed graph but be unbounded. Many unbounded operators on Hilbert space (e.g., differentiation on $L^2$) have closed graphs yet are not defined on the whole space.

Let $H$ be a Hilbert space. For every bounded linear functional $\ell: H \to \mathbb{R}$ there exists a unique $y \in H$ such that $\ell(x) = \langle x, y \rangle$ for all $x \in H$, and $\|\ell\|_{H^*} = \|y\|_H$. In particular, the map $y \mapsto \langle \cdot, y \rangle$ is an isometric isomorphism $H \to H^*$.

Hilbert spaces are self-dual: every continuous linear functional is "inner product with something." This is what makes Hilbert spaces the most tractable infinite-dimensional setting. The representer $y$ is obtained by projecting onto the orthogonal complement of $\ker \ell$.

This is the theorem that makes RKHS theory work: since evaluation functionals $\delta_x$ are bounded in an RKHS, there exists $k_x \in H$ with $f(x) = \langle f, k_x \rangle$, and $k(x, x') = \langle k_x, k_{x'} \rangle$ is the reproducing kernel. Riesz also gives the orthogonal projection, which is the foundation of least squares and conditional expectation.

This is the Hilbert space Riesz theorem. The Riesz-Markov theorem (which identifies $C_c(X)^*$ with regular Borel measures) is a different statement and applies in a different setting. In non-Hilbert Banach spaces, there is no canonical representation of the dual: $(L^p)^* = L^q$ requires $1 \leq p < \infty$, and $(L^\infty)^*$ is strictly larger than $L^1$.

Let $H$ be a Hilbert space and $a: H \times H \to \mathbb{R}$ a bilinear form satisfying:

1. Boundedness: $|a(u, v)| \leq M \|u\| \|v\|$ for some $M > 0$
2. Coercivity: $a(u, u) \geq \alpha \|u\|^2$ for some $\alpha > 0$

Then for every bounded linear functional $\ell \in H^*$ there exists a unique $u \in H$ such that $a(u, v) = \ell(v)$ for all $v \in H$, and $\|u\| \leq \alpha^{-1} \|\ell\|_{H^*}$.

Lax-Milgram is the natural extension of Riesz representation from symmetric inner products to non-symmetric coercive bilinear forms. When $a$ is symmetric, $a(u, v)$ defines an equivalent inner product on $H$ and the result reduces to Riesz. The non-symmetric case covers most variational formulations of elliptic PDEs, where the bilinear form $a(u, v) = \int \nabla u \cdot A \nabla v + b \cdot \nabla u\, v$ typically fails symmetry due to the convection term.

Lax-Milgram is the workhorse existence theorem for the Galerkin method and finite-element analysis: weak formulations of elliptic PDEs satisfy the boundedness and coercivity hypotheses, so Lax-Milgram guarantees a unique weak solution. In ML, it underpins Physics-Informed Neural Networks (PINNs) and neural Galerkin methods: the variational problem $\min_u a(u, u) - 2 \ell(u)$ is well-posed precisely when Lax-Milgram applies. The coercivity constant $\alpha$ controls the conditioning of the resulting linear system.

Coercivity is essential. Forms that are merely non-degenerate (e.g., $a(u, v) = \langle u, J v \rangle$ for a skew operator $J$) admit unique solutions only under additional inf-sup conditions (the Banach-Necas-Babuska theorem generalizes Lax-Milgram to this setting). Indefinite problems (Helmholtz at high frequency, saddle-point systems from constrained optimization) need the more general theory.