Model Merging and Weight Averaging

Given two models with weights $\theta_1$ and $\theta_2$, the simplest merge is linear interpolation:

$\theta_{\text{merged}} = \alpha \theta_1 + (1 - \alpha) \theta_2$

where $\alpha \in [0, 1]$ controls the mixing ratio. When $\alpha = 0.5$, this is a uniform average. This can be extended to $k$ models:

$\theta_{\text{merged}} = \sum_{i=1}^{k} \alpha_i \theta_i, \quad \sum_i \alpha_i = 1$

Spherical linear interpolation treats weight vectors as points on a hypersphere and interpolates along the great circle:

$\text{SLERP}(\theta_1, \theta_2, \alpha) = \frac{\sin((1-\alpha)\Omega)}{\sin\Omega} \theta_1 + \frac{\sin(\alpha\Omega)}{\sin\Omega} \theta_2$

where $\Omega = \arccos\left(\frac{\theta_1 \cdot \theta_2}{\|\theta_1\| \|\theta_2\|}\right)$ is the angle between the weight vectors.

Domain restriction. The formula requires $\Omega \in (0, \pi)$, i.e. $\theta_1$ and $\theta_2$ are neither parallel nor antiparallel. When $\Omega \to 0$ (nearly parallel), $\sin\Omega \to 0$ and the formula is numerically unstable. In that regime, fall back to linear interpolation (LERP), which is the limit of SLERP as $\Omega \to 0$.

Norm preservation. SLERP preserves the norm only when $\|\theta_1\| = \|\theta_2\|$ (typically both are normalized to unit vectors, or both layers have been rescaled to a common norm). For general vectors with different norms, SLERP interpolates direction on the sphere but does not automatically preserve magnitude.

Why SLERP over linear interpolation? For unit-norm weight vectors, linear interpolation shrinks the result's norm: $\|\alpha \theta_1 + (1-\alpha)\theta_2\| \leq 1$ with equality only when the vectors are parallel. SLERP preserves unit norm along the sphere. For neural network weights where the scale of activations matters, this can produce better results than linear interpolation.

Task Arithmetic (Ilharco et al., 2023) treats the difference between a fine-tuned model and its base as a portable "task vector":

$\tau_i = \theta_i - \theta_0$

Multiple task vectors can be composed linearly to add, subtract, or scale capabilities:

$\theta_{\text{merged}} = \theta_0 + \sum_i \lambda_i \tau_i$

Positive $\lambda_i$ adds a task capability, negative $\lambda_i$ subtracts (forgets) it, and scaling $\lambda_i$ controls strength. This is the algebraic scaffolding that TIES and DARE refine by first sparsifying or sign-resolving the $\tau_i$.

Matena and Raffel (2022) weight each parameter by the diagonal of the Fisher information matrix $F_i$ of model $i$, which estimates how much the model's output distribution changes when that parameter is perturbed:

$\theta_{\text{merged}} = \frac{\sum_i F_i \odot \theta_i}{\sum_i F_i}$

Parameters that matter more for a model's task (higher Fisher diagonal) get more weight in the merge. This improves over uniform averaging when the models disagree sharply about which parameters are important.

Jin et al. (2023) cast merging as a closed-form least-squares problem per linear layer. For a linear layer with input activations having second-moment matrix $G_i = \mathbb{E}[x x^\top]$ on model $i$'s task, the merged weight matrix solves:

$W_{\text{merged}} = \left(\sum_i G_i\right)^{-1} \sum_i G_i W_i$

This is "dataless" in the sense that the $G_i$ are precomputed statistics, not a training loop. RegMean outperforms Fisher-weighted merging when the input distributions across models differ substantially.

TIES-Merging (Yadav et al., 2023) addresses a problem with naive averaging: when merging $k$ models, task-specific weight changes from different models can cancel each other out if they point in opposite directions.

The TIES algorithm has three steps:

1. Trim: For each model, compute the task vector $\tau_i = \theta_i - \theta_0$ (the change from the base model). Keep the top-k% of entries by absolute magnitude (Yadav et al. use $k = 20\%$ as the default) and zero out the rest. This is a per-model top-k selection, not a fixed threshold.

2. Elect sign: For each weight position, take a magnitude-weighted majority vote across models on whether the change should be positive or negative. This resolves sign conflicts.

3. Disjoint merge: For each position, average only the entries whose sign matches the elected sign. Positions with no matching entries are set to zero.

$\theta_{\text{TIES}} = \theta_0 + \lambda \cdot \text{disjoint\_merge}(\tau_1, \ldots, \tau_k)$

where $\lambda$ is a scaling factor.

DARE (Yu et al., 2024) takes a more aggressive approach to sparsification before merging:

1. Compute task vectors $\tau_i = \theta_i - \theta_0$ for each model.
2. For each task vector, randomly drop a fraction $p$ of the entries (set them to zero). DARE uses high drop rates, typically $p \in [0.9, 0.99]$.
3. Rescale the remaining entries by $1/(1-p)$ to preserve the expected magnitude.
4. Merge the sparsified task vectors by averaging.

$\tau_i^{\text{DARE}} = \frac{1}{1-p} \cdot m_i \odot \tau_i$

where $m_i$ is a binary mask with entries drawn i.i.d. Bernoulli($1-p$).

Why this works: DARE's premise is that fine-tuning task vectors are highly redundant. Most entries contribute little to the task-specific capability, so dropping 90-99% of them and rescaling preserves the expected contribution while sharply reducing interference between models during merging. Yu et al. show that drop rates up to $p = 0.99$ retain most of the fine-tuned capability on language tasks.

Akiba et al. (2024, Sakana AI) search for per-layer merge coefficients using evolutionary optimization (CMA-ES). The search space combines two axes:

1. Parameter space (PS) merging: per-layer mixing weights $\alpha_{\ell}$ for each layer $\ell$, applied to task vectors or SLERP.
2. Data-flow space (DFS) merging: a discrete per-layer routing that selects which source model's layer to use at each depth.

The fitness is a downstream-task evaluation on a held-out set. This removes the need to hand-tune $\lambda$ and per-layer weights, at the cost of requiring evaluation compute during the search.

Ainsworth, Hayase, and Srinivasa (2023) formalize the permutation symmetry of neural networks: for a model with permutation $\pi_{\ell}$ applied to the hidden units of layer $\ell$, there is an equivalent model obtained by permuting the rows of $W_{\ell}$ and the columns of $W_{\ell+1}$ by $\pi_{\ell}$, giving identical input-output behaviour. Two independently trained models $\theta_1, \theta_2$ occupy weight-space points related by (approximately) such a permutation.

Weight matching and activation matching algorithms search for permutations $\{\pi_{\ell}\}$ that align $\theta_2$ to $\theta_1$. After alignment, the linear interpolation $\alpha \theta_1 + (1-\alpha) \pi(\theta_2)$ stays in a low-loss region on vision benchmarks, recovering linear mode connectivity for independently trained models. Related follow-ups (ZipIt!, Stoica et al. 2023; Model Breadcrumbs, Davari and Belilovsky 2024) extend this idea to merging models without a shared base or to sparsifying task vectors for more robust composition.

"Model soups" (Wortsman et al., 2022) refers to averaging the weights of models trained with different hyperparameters (learning rate, weight decay, augmentation) on the same task. Instead of selecting the best model by validation performance, you average all models that exceed a quality threshold.

The result consistently outperforms the best individual model because:

• Each hyperparameter setting explores a slightly different part of the basin
• Averaging moves toward the center of the basin (similar to SWA)
• The averaged model is more robust to distribution shift than any individual model