mergekit

Tools for merging pretrained large language models.

7,065 stars Python #llama#llm#model-merging
RAW Doc

Create A Merge Method

Extending MergeKit with Custom Merge Methods

Overview

MergeKit offers two different paths for implementing custom merge methods:

Decorator APIClass-based API
Complexity

Simple function-based | Full class implementation |
| Abstraction Level | Higher-level | Lower-level |
| Parameter Handling | Automatic validation | Manual configuration |
| Execution Flow | Single function | Arbitrary computation graph |
| Best For | Most merge methods | Complex multi-stage, multi-input strategies |

Either approach benefits from MergeKit's underlying task system for resource management and execution control. The question of which to use largely depends on the complexity of the merge operation and the level of control needed.

Note on Parameter Configuration: MergeKit uses a hierarchical YAML-based configuration system. Parameters for your custom merge methods (both scalar and per-model) can be defined at various levels (e.g., globally, per-model, per-slice). The values your merge function or task receives are resolved by MergeKit based on this hierarchy and context. For full details on configuration structure and parameter precedence, please refer to the Merge Configuration section of the README.

Core Task System Features

MergeKit's computational graph infrastructure provides sophisticated resource management that all merge methods inherit:

  • Smart Memory Management

- Automatic return value lifecycle tracking
- Early value eviction when no longer needed
- Optimized shard loading based on task groups

  • Device Management

- Automatic tensor movement between compute and storage devices
- Support for both CPU and GPU execution

  • Task Scheduling

- Tasks grouped by tensor shard to minimize memory usage
- Loads deferred until last possible moment (via priority system)
- Execution ordered to optimize shard residency

Decorator API

Best for straightforward merge operations that can be expressed as a single tensor transformation. Features:

  • Parameter validation, type checking, and value resolution
  • Configuration schema generation
  • Simplified base model handling
  • Default GPU acceleration opt-in

Class-based API

Choose when you need:

  • Multi-stage merge operations
  • Custom computation graphs
  • Direct access to weight metadata
  • Complex parameter types
  • Fine-grained control over execution

Decorator API Implementation

Basic Workflow

  1. Define a type-annotated Python function with your merge logic
  2. Add the @merge_method decorator with configuration
  3. Ensure the module containing your function is imported

- For MergeKit to discover your decorated merge method, the Python module containing it must be imported during MergeKit's initialization. Add an import statement for your module in mergekit/merge_methods/__init__.py. Once the module is imported, the @merge_method decorator handles the registration of the method with MergeKit.

Example: Weighted Average

python
from mergekit.merge_methods.easy_define import merge_method
from typing import List
import torch

@merge_method(
    name="weighted_average",
    pretty_name="Weighted Average",            # Optional: human-readable name
    reference_url="https://example.com/docs",  # Optional: documentation or paper link
)
def average_merge(
    tensors: List[torch.Tensor],  # Required: input tensors
    weight: List[float],          # Vector parameter (one float per model)
    normalize: bool = True,       # Scalar parameter with default
) -> torch.Tensor:
    if normalize:
        total = sum(weight)
        weight = [w / total for w in weight]

    return sum(t * w for t, w in zip(tensors, weight))

This enables configurations like:

yaml
merge_method: weighted_average
models:
  - model: model1
    parameters:
      weight: 0.3
  - model: model2
    parameters:
      weight: 0.7
parameters: # Global parameters
  normalize: true

Parameter Types and Handling

The decorator supports three parameter categories:

  1. Scalar Parameters

- Types: bool, float, or int
- Single value for all models
- Without defaults they become required parameters
- Example: normalize: bool = True

  1. Vector Parameters

- Types: List[float] or List[int] only
- Configured per-model
- Default values must be single numbers, not lists, as they are broadcasted
- Example: weights: List[float]

  1. Base Model Integration

The tensors: List[torch.Tensor] argument and an optional base_tensor argument in your function signature interact with the base_model specified in the YAML configuration as follows:

- If your function includes a base_tensor parameter (e.g., base_tensor: torch.Tensor or base_tensor: Optional[torch.Tensor]):
- The base_tensor argument will receive the tensor from the base_model specified in the YAML. If annotated as Optional and no base_model is configured, it will be None.
- The tensors: List[torch.Tensor] argument will only contain tensors from the models specified under the models: key in the YAML, in order. It will not include the base model's tensor.
- If your function does not include a base_tensor parameter:
- If a base_model is specified in the YAML, its tensor will be the first element in the tensors: List[torch.Tensor] list (i.e., tensors[0]).
- Subsequent elements (tensors[1:]) will correspond to the models listed under the models: key in the YAML, in order.
- If no base_model is specified, the tensors list will directly correspond to the models listed under the models: key.

  1. Special Auto-Populated Parameters

Certain parameter names in your function signature have special meaning and are auto-populated by MergeKit if present. You do not configure these directly in the YAML parameters sections for your method; MergeKit provides them.
- output_weight: WeightInfo: If your function accepts an argument named output_weight annotated with WeightInfo, MergeKit will pass metadata about the specific weight tensor being computed.
- base_model: ModelReference (or Optional[ModelReference]): If your function accepts base_model annotated with ModelReference, MergeKit will pass a reference to the base model if one is used in the configuration for this merge operation.

Class-based API Implementation

For complex merges requiring granular control, implement MergeMethod and Task classes:

Example Implementation

text
/* Detailed source-code truncated for AI context efficiency. */

Task Scheduling System

The class-based API provides fine-grained control over execution:

  • Priority Control: Override priority() to influence execution order within groups
  • Task Grouping: Use group_label() to batch similar operations
  • Resource Management:

- Automatic tensor lifecycle tracking
- Memory optimization via early tensor eviction
- Smart device placement for computation vs storage

  • Computation Graph: Build complex flows by connecting multiple tasks

Implementation Requirements

  1. Task Class:

- Must implement execute() with proper type annotations
- Must implement arguments() to declare dependencies
- Optionally override priority(), group_label(), uses_accelerator()

  1. Method Class:

- Must implement core methods: name(), make_task()
- Optional methods: pretty_name(), reference_url()
- Define parameters via parameters() and tensor_parameters()

Registration

Add class-based methods to STATIC_MERGE_METHODS in mergekit/merge_methods/registry.py:

python
from mergekit.merge_methods.my_module import CustomMerge

STATIC_MERGE_METHODS: List[MergeMethod] = [
    CustomMerge(),
    # other methods...
]

Reference Implementations

  1. Linear Merge (mergekit.merge_methods.linear):

- Basic weighted averaging
- Good example of class-based implementation

  1. Multi-SLERP (mergekit.merge_methods.multislerp):

- Hypersphere interpolation
- Complex decorator usage example

  1. Task Arithmetic (mergekit.merge_methods.task_arithmetic):

- Advanced graph-based implementation
- TIES/Magnitude pruning example

Evolve

mergekit-evolve

mergekit-evolve is a script that uses an evolutionary algorithm (CMA-ES) to optimize the parameters of a merge against model metrics. This is inspired by SakanaAI's Evolutionary Optimization of Model Merging Recipes, in particular their parameter-space approach. mergekit-evolve uses EleutherAI's Language Model Evaluation Harness to define and evaluate the scoring function. The script is set up to be run either single-node or on a Ray cluster and has a few different strategies for scheduling operations depending on your particular configuration of compute.

Installation

Install mergekit with the evolve (and optionally vllm) features:

sh
git clone https://github.com/arcee-ai/mergekit.git
cd mergekit

pip install -e .[evolve,vllm]

If you had a perfectly good pytorch environment going and installing an older version of vLLM downgraded it and broke flash attention, run the following commands to fix it:

sh
pip uninstall flash-attn
pip cache purge
pip install flash-attn

Configuration

mergekit-evolve takes in a YAML configuration file that defines how the merge is parameterized and what metrics to optimize. The general syntax is as follows:

yml
genome:
    models:
       - model_1
       - model_2
       ...
       - model_n
    merge_method: dare_ties
    base_model: base_model_if_needed
    tokenizer_source: null # optional
    layer_granularity: 8

    # optional:
    normalize: false
    allow_negative_weights: false
    smooth: false
    filters: ...
tasks:
  - name: lm_eval_task_name
    weight: 1.0 # optional
    metric: "acc,none" # defaults to acc,none
  - name: ... # as many as you want

Genome Definition

The genome section of the configuration file defines the parameter space that mergekit-evolve will be optimizing in.

`models`

This should be a list of all of the models you want available to be merged. Depending on the merge method not all are guaranteed to be used in the final merge.

`merge_method`

Merge method to be used. Currently supported values are linear, dare_ties, task_arithmetic, ties, and slerp.

`base_model`

The base model for the merge, if applicable.

`layer_granularity`

A set of parameters will be introduced for each consecutive slice of layer_granularity layers. So for example, a 32-layer model like mistralai/Mistral-7B-v0.1 with layer_granularity: 8 will be divided into 4 groups of 8 layers with different merge parameters for each. The value specified here must be a divisor of the number of layers in your input models. Large values of layer_granularity will reduce the search space greatly, meaning you will get faster convergence at the cost of a potentially less good global solution.

When not set, one set of parameters will be used for all layers.

`normalize`

Sets the normalize flag when merging. For methods like linear, ties, and dare_ties this constrains the search space to a set of definitely valid models. Similarly to layer_granularity, this can greatly speed up convergence at the cost of ruling out oddball solutions that might score better than more standard merges.

`allow_negative_weights`

Pretty self explanatory. When this flag is not set, the absolute value of weight parameters is used. Sensible search space reduction for linear and slerp. For task arithmetic based methods you probably want allow_negative_weights: true.

`smooth`

If set to true, then parameter values will be interpolated across layers instead of assigning a single, fixed value to each block.

`filters`

Accepts a list of filters, as in mergekit-yaml, by which to separate the parameters. So, for example, setting filters as below for a Llama-based merge:

yaml
filters:
  - self_attn
  - mlp

Will divide up the merge parameters into three groups - self attention parameters, MLP parameters, and a third for everything else. Separating the parameters out like this can be very beneficial when merging models trained on different prompt formats. It also makes your parameter space three times as big though!

Task Definition

To evaluate the produced merges you need to specify a list of tasks supported by the EleutherAI LM evaluation harness. This can be either built in tasks (don't be naughty) or tasks you define yourself (see the New Task Guide for how). If your task does not use acc as the metric then you must specify the correct metric name. Each task can also optionally have a weight associated.

mergekit-evolve aims to maximize the score of the merge, so if you are using any tasks or metrics where a lower score is better (like perplexity) be sure to assign a negative weight to that task.

Running `mergekit-evolve`

sh
mergekit-evolve [OPTIONS] --storage-path PATH GENOME_CONFIG_PATH

mergekit-evolve needs a storage path specified, where it will save the input models, merges to evaluate, and the config for the current best merge evaluated. If you are not using in-memory merging this can require a _lot_ of space - expect at least one fp16 model per GPU.

Some important options:

Scheduling Strategy (`--strategy`)

There are three different strategies implemented for scheduling merging and evaluation jobs.

`pool`

Assigns an actor to each GPU in your cluster and guarantees merges and evaluations are performed on the same node. This is a safe default suitable for any configuration, local or distributed.

`buffered`

Maintains a buffer of tasks scheduled to ensure that there is always a model merging or ready to evaluate for each GPU. Allows for concurrent merging and evaluation of models on the same GPU if enough VRAM is available. Only suitable for a single-node setup or when --storage-path points to a fast shared filesystem.

`serial`

Uses Ray placement groups to ensure merges and their evaluations happen on the same node, but otherwise just lets Ray take the wheel. Maybe give a try if you're having trouble with the other two, otherwise probably don't use it.

Evaluation LLM Backend

By default mergekit-evolve will use the hf backend for lm-eval. To use vLLM instead, pass the --vllm flag.

On-Disk vs. In-Memory

By default mergekit-evolve will perform merges, write the result to disk, then start up an instance of lm-eval pointing at that path. This is a safe default and will generally always work but also causes a lot of GPU downtime and eats disk space. When using the pool scheduling strategy, you have the option to instead keep a model resident in memory and directly update its parameters instead of merging to disk. This is much faster and uses no additional disk space. However, it does involve mucking around in the internals of vLLM and the LM evaluation harness. So it might break at any moment! Choose wisely. Use --in-memory to enable this mode.

Task search path

If you're using custom task definitions (and you should be) then you can append to the search path using the --task-search-path option. This should point to the directory your custom task YAML is in (or a parent of that directory). Multiple paths can be included by repeating the option.

Batch size

Override the batch size used during merge evaluation. If using vLLM auto is recommended (default).

CMA-ES options

`--max-fevals`

Maximum number of merges to evaluate. Note that the cma package is very loosey-goosey with this number and will happily go over by 50% depending on the size of each generation. Set to 100 by default.

`--sigma0`

Initial value of sigma for CMA-ES. No need to play with this unless you really know what you're doing.

WandB logging

mergekit-evolve supports logging metrics to Weights & Biases. Enable this functionality with the --wandb flag. Project and entity names can be overridden with the --wandb-project and --wandb-entity options.

Example

sh
mergekit-evolve --strategy pool --wandb --wandb-project mergekit-evolve --wandb-entity arcee-ai --storage-path /path/to/mergekit-evolve/ ./config.yml

Output

mergekit-evolve will write the merge configuration for the best merge found so far to the storage path with the filename best_config.yaml. If you're using WandB it will also log the config as an artifact. The script will keep running until a KeyboardInterrupt is received or --max-fevals is generously exceeded.

Caveats

mergekit-evolve is a work in progress and has probably not been tested on your specific configuration. Keep an eye on the output before leaving it running, and if you run in to any issues don't hesitate to file an issue!

Acknowledgements

Thanks to SakanaAI for the inspiration and the EleutherAI team for the LM evaluation harness.

Merge Methods

Merge Method Guide

Table of Contents

- Linear (linear)

- SLERP (slerp)
- NuSLERP (nuslerp)
- Multi-SLERP (multislerp)
- Karcher Mean (karcher)

- Task Arithmetic (task_arithmetic)
- TIES-Merging (ties)
- DARE (dare_linear, dare_ties)
- DELLA (della, della_linear)
- Model Breadcrumbs (breadcrumbs, breadcrumbs_ties)
- SCE (sce)
- RAM (ram, ramplus_tl)

- Model Stock (model_stock)
- Nearswap (nearswap)
- Arcee Fusion (arcee_fusion)
- Passthrough (passthrough)

Overview

This guide provides detailed information about the various model merging algorithms available in mergekit. Each method has specific use cases, parameters, and applications for combining machine learning models.

Basic Merging Methods

Linear (`linear`)

Concept: Computes a simple weighted average of the parameters from the input models. This is one of the most basic and widely used merging techniques.

Use Cases:

  • Averaging multiple checkpoints of the same fine-tuning run ("model soups")
  • Combining models with very similar architectures and training data
  • Simple ensemble-like behavior in a single model

Inputs: Takes 2 or more models. No base_model is typically used.

Key Parameters:

  • weight (per-model): The contribution of each model to the average
  • normalize (global): If true (default), weights are normalized to sum to 1

Reference: Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time

Spherical Interpolation Methods

SLERP (`slerp`)

Concept: Performs Spherical Linear Interpolation in the weight space between two models. This creates a path along a hypersphere, ensuring the interpolated model maintains a similar "norm" or "magnitude" to the original models.

Use Cases:

  • Creating smooth transitions or intermediate points between two distinct models
  • Exploring the space between two models with potentially different capabilities

Inputs: Requires exactly 2 models. One model must be specified as base_model.

Key Parameters:

  • t (global): Interpolation factor. t=0 yields the base_model, t=1 yields the other model

Reference: Wikipedia: Slerp

NuSLERP (`nuslerp`)

Concept: An enhanced version of SLERP offering more flexible configuration and faster execution. It allows SLERP between two models directly. If a base_model is provided, NuSLERP calculates task vectors (the difference between each of the two main models and the base model) and then performs SLERP on these task vectors before adding the result back to the base_model.

Use Cases:

  • Similar to SLERP, but with more control over weighting for the two primary models.
  • To replicate the behavior of the original slerp method (if base_model is not used, and weights are set to 1-t for the first model and t for the second).
  • To perform SLERP on task vectors when a base_model is provided, allowing for interpolation of model changes relative to a common ancestor.

Inputs: Requires exactly 2 models. A base_model can optionally be provided (it must be distinct from the two main models).

Key Parameters:

  • weight (per-model): Relative weighting for each of the two main models. These are used to calculate the interpolation factor t (where t = model2_weight / (model1_weight + model2_weight)).
  • nuslerp_flatten (global): If false, performs row/column-wise interpolation. Default true
  • nuslerp_row_wise (global): If true (and nuslerp_flatten is false), SLERPs row vectors instead of column vectors. Default false

Multi-SLERP (`multislerp`)

Concept: Implements barycentric interpolation on a hypersphere for more than two models. It projects points onto a tangent space at their weighted Euclidean mean, performs interpolation, and projects back.

Use Cases:

  • Creating a spherical average of multiple models
  • Finding a central point in the weight space of several related models

Inputs: Takes 2 or more models. A base_model can optionally be provided to operate in task vector space.

Key Parameters:

  • weight (per-model): Relative weighting for each model
  • normalize_weights (global): If true (default), weights are normalized
  • eps (global): Small constant for numerical stability. Default 1e-8

Karcher Mean (`karcher`)

Concept: Computes the Karcher mean (also known as the Riemannian barycenter or Fréchet mean) of the input model parameters. This provides a geometrically sound way to average points on a manifold, which is suitable for model weights.

Use Cases:

  • Finding a "central" or "average" model among a set of diverse models in a way that respects the geometry of the weight space
  • More robust averaging than simple linear averaging, especially for models far apart in weight space

Inputs: Takes 2 or more models. No base_model is used.

Key Parameters:

  • max_iter (global): Maximum iterations for the Karcher mean algorithm. Default 10
  • tol (global): Convergence tolerance. Default 1e-5

Reference: Functionality-Oriented LLM Merging on the Fisher-Rao Manifold

Task Vector Methods

The following methods build upon the concept of "task vectors," which represent the difference between a fine-tuned model and a base model.

Task Arithmetic (`task_arithmetic`)

Concept: Computes "task vectors" for each model by subtracting a base_model. These task vectors are then combined as a weighted average and added back to the base_model.

Use Cases:

  • Combining skills from multiple models fine-tuned from a common ancestor
  • Transferring specific capabilities (e.g., coding ability, instruction following) from one model to another
  • Steering style or behavior of a model by adding small task vectors from other models

Inputs: Requires a base_model and one or more other models.

Key Parameters:

  • weight (per-model): Weight for each model's task vector in the merge
  • lambda (global): Scaling factor applied to the summed task vectors before adding back to the base. Default 1.0

Reference: Editing Models with Task Arithmetic

TIES-Merging (`ties`)

Concept: Builds on Task Arithmetic by sparsifying task vectors and applying a sign consensus algorithm. This helps to resolve interference when merging multiple models and retain more of their individual strengths.

Use Cases:

  • Merging a larger number of models effectively
  • Reducing parameter interference and negative synergy between merged models

Inputs: Requires 2 or more models, plus one base_model.

Key Parameters:

  • weight (per-model): Weight for each model's task vector
  • density (per-model): Fraction of weights to retain in each sparsified task vector
  • lambda (global): As in Task Arithmetic

Reference: TIES-Merging: Resolving Interference When Merging Models

DARE (`dare_linear`, `dare_ties`)

Concept: Similar to TIES, DARE sparsifies task vectors to reduce interference. However, DARE uses random pruning with a novel rescaling technique to better match the performance of the original models.

Variants:

  • dare_linear: DARE pruning without the TIES sign consensus
  • dare_ties: DARE pruning with the TIES sign consensus

Use Cases:

  • Robustly combining multiple fine-tuned models, often yielding better performance than TIES in some scenarios

Inputs: Requires 2 or more models, plus one base_model.

Key Parameters:

  • weight (per-model): Weight for each model's task vector
  • density (per-model): Fraction of weights to retain after random pruning
  • lambda (global): As in Task Arithmetic
  • rescale (global, for dare_linear): If true (default), applies DARE's rescaling

Reference: Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

DELLA (`della`, `della_linear`)

Concept: Extends DARE by using adaptive pruning based on parameter magnitudes within each row of the delta parameters (task vectors). It calculates keep probabilities for each parameter: parameters with larger magnitudes within a row are assigned higher probabilities of being kept, while parameters with smaller magnitudes are assigned lower probabilities. These keep probabilities are scaled to range from density - epsilon (for the smallest magnitude element in a row) to density + epsilon (for the largest magnitude element in a row). This method aims to retain important changes while reducing interference, followed by DARE-like rescaling.

Variants:

  • della: DELLA pruning with TIES sign consensus
  • della_linear: DELLA pruning without TIES sign consensus

Use Cases:

  • Fine-grained control over pruning by prioritizing parameters with larger magnitude changes
  • Combining models where preserving the most significant changes is crucial

Inputs: Requires 2 or more models, plus one base_model.

Key Parameters:

  • weight (per-model): Weight for each model's task vector
  • density (per-model): Target fraction of weights to retain in differences from the base model
  • epsilon (per-model): Defines the half-width of the range for keep probabilities. Keep probabilities for parameters in a row will range from density - epsilon to density + epsilon, mapped from the smallest to largest magnitude parameters in that row, respectively. epsilon must be chosen such that density - epsilon > 0 and density + epsilon < 1.
  • lambda (global): As in Task Arithmetic

Reference: DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling

Model Breadcrumbs (`breadcrumbs`, `breadcrumbs_ties`)

Concept: An extension of task arithmetic designed to sparsify task vectors by pruning parameters with both the smallest and the largest absolute magnitudes (often considered outliers). This method operates in two main steps on the task vector (the difference between a fine-tuned model and the base_model):

  1. First, a gamma fraction of the parameters with the largest absolute magnitudes are identified for removal.
  2. Then, parameters with the smallest absolute magnitudes are identified for removal. The quantity of these smallest parameters to remove is determined such that the final density of parameters retained in the task vector is achieved, after accounting for the largest ones removed.

The intention is to isolate and merge the "meaty," mid-range magnitude changes from the task vector, potentially filtering out noise (smallest changes) and overly dominant or conflicting large changes (largest changes).

Variants:

  • breadcrumbs: Model Breadcrumbs pruning without TIES sign consensus
  • breadcrumbs_ties: Model Breadcrumbs pruning with TIES sign consensus

Use Cases:

  • Merging models where extreme parameter changes might be detrimental or noisy
  • Refining task vectors by focusing on mid-range modifications, removing both the least significant and most extreme changes

Inputs: Requires 2 or more models, plus one base_model.

Key Parameters:

  • weight (per-model): Weight for each model's task vector.
  • gamma (per-model): The fraction of parameters with the largest absolute magnitudes in the task vector to be pruned (removed). For example, a gamma of 0.01 targets the removal of the top 1% of parameters with the highest absolute values. This parameter corresponds to β (beta) as described in the reference paper.
  • density (per-model): The final target fraction of parameters to retain in the task vector after both pruning steps (removal of largest gamma fraction and a corresponding fraction of smallest magnitude parameters).

- The fraction of parameters with the smallest absolute magnitudes that will be pruned is calculated based on density and gamma. Specifically, it is max(0, 1.0 - density - gamma).
- Example: If density: 0.9 and gamma: 0.01:
- The top 0.01 (1%) largest magnitude parameters are removed.
- The bottom 1.0 - 0.9 - 0.01 = 0.09 (9%) smallest magnitude parameters are also removed.
- This results in 0.9 (90%) of the parameters being retained.
- Edge Case: If gamma is set high enough such that gamma >= 1.0 - density (meaning 1.0 - density - gamma <= 0), then the number of largest magnitude parameters actually pruned will be adjusted to 1.0 - density, and no smallest magnitude parameters will be pruned (i.e., the fraction of smallest parameters pruned becomes 0). This ensures the density target is always respected and represents the fraction of parameters kept.

  • lambda (global): As in Task Arithmetic.

Reference: Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks

SCE (`sce`)

Concept: The SCE (Select, Calculate, Erase) method performs adaptive matrix-level merging. It first computes task vectors (differences from the base_model). Then, it follows a three-step process for each parameter matrix (tensor):

  1. Select (Variance-Based Masking): Optionally, parameter positions that show low variance across the different models' task vectors are identified and zeroed out. This is controlled by the select_topk parameter. If select_topk < 1.0, only the top select_topk fraction of parameter positions with the highest variance are kept active in the task vectors for subsequent steps.
  2. Calculate (Weighting): Matrix-level merging coefficients (weights) are calculated for each model's task vector. These weights are derived from the mean of the squares of the elements within each task vector and are normalized across the models.
  3. Erase (Sign Consensus): The sign-consensus algorithm from TIES is applied to the task vectors.

Finally, the (variance-selected, calculated-weighted, and sign-agreed) task vectors are summed together, normalized by the sum of the effective applied weights at each position, and then added back to the base_model.

Use Cases:

  • Dynamically weighting the contribution of different models at the matrix level based on parameter variance and calculated importance
  • Useful when some models contribute more significantly or consistently to certain parameter matrices than others
  • Merging models by focusing on high-variance, consistently signed changes

Inputs: Requires 2 or more models, plus one base_model.

Key Parameters:

  • select_topk (global): The fraction of parameter positions to retain based on their variance values across the different input models' task vectors. For each parameter position, variance is calculated across all task vectors. Only positions corresponding to the select_topk fraction with the highest variances are kept (i.e., their values in all task vectors are preserved for the next steps). Positions with lower variance are zeroed out in all task vectors. Set to 1.0 (default) to disable this variance-based selection step. This corresponds to τ (tau) in the reference paper.

Reference: FuseChat: Knowledge Fusion of Chat Models

RAM (`ram`, `ramplus_tl`)

Concept: Reinforced Agent Merging (RAM) is a task vector method specifically designed for merging reinforcement learning (RL)-trained agents. The key insight is that on-policy RL induces task vectors that are highly sparse and heterogeneous, unlike the denser vectors from supervised fine-tuning. Traditional averaging methods dilute critical task-specific behaviors in this setting.

RAM addresses this by classifying each parameter into three categories based on how many models modify it:

  • Inactive: No models modify this parameter (magnitude below epsilon)
  • Unique: Exactly one model modifies this parameter
  • Shared: Multiple models modify this parameter

For inactive parameters, the base model value is preserved. Unique contributions are summed directly (preserving task-specific behaviors), while shared contributions are averaged (resolving conflicts). This selective merging strategy prevents the dilution of specialized agent behaviors.

Variants:

  • ram: The base RAM method without rescaling. Unique parameters are preserved as-is, shared parameters are averaged.
  • ramplus_tl: RAM+ extends RAM by applying an adaptive rescaling factor λ to unique contributions, compensating for signal dilution caused by averaging shared parameters. The rescaling formula is: λ = 1 + r · clip(ρ, 0, α), where ρ is the ratio of shared to unique parameter counts for each model within each tensor. mergekit implements a tensor-local variant, in which the overlap-unique ratio ρ is computed per-tensor rather than globally.

Use Cases:

  • Scenarios where task vectors are sparse and heterogeneous (common in RL fine-tuning)
  • Merging multiple RL-trained agents from different tasks into a single generalist model

Inputs: Requires a base_model and one or more other models.

Key Parameters:

  • epsilon (global): Threshold for determining if a parameter is modified (values with absolute magnitude at or below epsilon are treated as inactive/unchanged). Default 1e-5

For ramplus_tl only:

  • r (global): Rescaling strength that controls the amplification of unique contributions. Higher values increase the reinforcement of unique parameters. Default 0.1
  • alpha (global): Stability bound that clamps the overlap-unique ratio ρ, limiting the maximum rescaling factor. Default 0.2

Reference: Reinforced Agent Merging

Specialized Methods

Model Stock (`model_stock`)

Concept: Uses geometric properties of fine-tuned models relative to a base_model to compute an optimized interpolation weight. It then performs a linear interpolation between the base_model and the average of the other input models using this computed weight. Specifically, task vectors (differences between other models and the base_model) are used to calculate pairwise cosine similarities. The average of these similarities informs the interpolation factor t. The final merged tensor is t average_of_other_models + (1 - t) base_model.

Use Cases:

  • Finding effective weights for linearly combining multiple models where one model serves as a clear reference (base_model).
  • When a more principled, data-driven approach to linear interpolation between a base and a group of variants is desired over manual weight tuning.
  • Particularly strong for combining different training runs that were fine-tuned from the same base_model over the same or similar datasets.

Inputs: Requires at least 3 models: one base_model and at least two other models.

Key Parameters:

  • filter_wise (global): If true, weight calculation is per-row rather than per-tensor (not generally recommended). Default false

Reference: Model Stock: All we need is just a few fine-tuned models

Nearswap (`nearswap`)

Concept: Interpolates the base model with parameters from a secondary model primarily where they are already similar. The interpolation strength towards the secondary model is inversely proportional to the absolute difference of their parameters, modulated by the t parameter. When the parameters are similar, the interpolation is stronger, and when they are different, it is weaker.

Use Cases:

  • Selectively pulling in similar parameters from a secondary model while preserving different parameters from the base model
  • Fine-grained parameter-wise merging that respects the existing structure of the base model

Inputs: Requires exactly 2 models. One model must be specified as base_model.

Key Parameters:

  • t (global): Controls the interpolation strength. Higher values increase the influence of the secondary model for similar parameters

Algorithm: For each parameter, computes weight = (t / |base - secondary|).clamp(0, 1), then returns weight secondary + (1 - weight) base

Reference: QuartetAnemoi-70B-t0.0001 on Hugging Face

Arcee Fusion (`arcee_fusion`)

Concept: Merges two models by dynamically identifying and fusing important parameter changes. It calculates importance scores based on parameter differences and KL divergence, then uses a dynamic threshold to create a fusion mask.

Use Cases:

  • Intelligently combining two models by prioritizing the most salient differences

Inputs: Requires exactly 2 models. One model must be specified as base_model.

Key Parameters: None beyond standard model selection

Reference: MergeKit v0.1 Release Blog

Passthrough (`passthrough`)

Concept: A no-op merge method that simply passes input tensors through unmodified from a single input model.

Use Cases:

  • Layer-stacking or "Frankenmerging" where you assemble a model from specific unmodified layer ranges or individual tensors from one or more "donor" models
  • Useful as a building block in more complex slices configurations

Inputs: Takes exactly 1 model.

Key Parameters:

  • scale (per-model, optional): A scalar to multiply the tensor by. Useful for scaling specific layers, e.g., {"filter": "down_proj", "value": 0.5}

Summary

Merge methods serve different purposes and have different design goals. The choice of method depends on your specific use case, including the number of models, their relationships, and the desired characteristics of the final merged model.

For beginners, starting with linear, nuslerp, or task_arithmetic can provide good results. For more advanced use cases, methods like ties, dare_ties, or della offer sophisticated ways to handle interference between multiple models while preserving their individual strengths. There is no "best" merge method; the right choice depends on your specific needs and the models you are working with, and selection is often more art than science. I encourage you to experiment with different methods and parameters vigorously to both find the best results for your use case and to learn more about how these methods work. Happy merging!

Contributing

If you have ideas for new merge methods or improvements to existing ones, we'd be glad to have you involved! Check out the Contributing Guide and Creating a Merge Method for more information on how to get started.

Moe

mergekit-moe

mergekit-moe is a script for combining Mistral or Llama models of the same size into Mixtral Mixture of Experts models. The script will combine the self-attention and layer normalization parameters from a "base" model with the MLP parameters from a set of "expert" models.

If using the hidden or cheap_embed gate mode, the output model will be usable without any further training. If you are initializing a model to do further training on, such as for sparse upcycling, then use the random gate mode to get a model ready for training.

Configuration

mergekit-moe uses its own YML configuration syntax, which looks like so:

yml
base_model: path/to/self_attn_donor
gate_mode: hidden # one of "hidden", "cheap_embed", or "random"
dtype: bfloat16 # output dtype (float32, float16, or bfloat16)
## (optional)
# experts_per_token: 2
experts:
  - source_model: expert_model_1
    positive_prompts:
      - "This is a prompt that is demonstrative of what expert_model_1 excels at"
    ## (optional)
    # negative_prompts:
    #   - "This is a prompt expert_model_1 should not be used for"
  - source_model: expert_model_2
  # ... and so on

The script takes two arguments, an input config and an output path: mergekit-moe ./config.yml ./my-clowncar-moe-12x180B

Currently the script can output models that use the Mixtral, Deepseek MoE, or Qwen MoE architectures. Some output architectures support a shared expert which will be activated for all tokens, which can be configured like this:

yml
base_model: path/to/self_attn_donor
gate_mode: hidden # one of "hidden", "cheap_embed", or "random"
dtype: bfloat16 # output dtype (float32, float16, or bfloat16)
experts:
  ...
shared_experts:
  - source_model: model_name
    positive_prompts: # required by Qwen MoE for "hidden" gate mode, otherwise not allowed
      - "blah blah"
    # (optional, but recommended:)
    residual_scale: 0.1 # downweight output from shared expert to prevent overcooking the model

Currently only up to one shared expert is supported.

An appropriate architecture will be inferred based on the input models and presence or absence of shared experts in your configuration. Alternatively, you can explicitly specify an output architecture by setting the architecture: field in your config. For example:

yml
base_model: path/to/self_attn_donor
architecture: qwen
# ... and so on

Gate Modes

There are three methods for populating the MoE gates implemented.

"hidden"

Uses the hidden state representations of the positive/negative prompts for MoE gate parameters. Best quality and most effective option; the default. Requires evaluating each prompt using the base model so you might not be able to use this on constrained hardware (depending on the model). You can use --load-in-8bit or --load-in-4bit to reduce VRAM usage.

"cheap_embed"

Uses only the raw token embedding of the prompts, using the same gate parameters for every layer. Distinctly less effective than "hidden". Can be run on much, much lower end hardware.

"random"

Randomly initializes the MoE gates. Good for if you are going to fine tune the model afterwards, or maybe if you want something a little unhinged? I won't judge.

Example Configurations

Sparse upcycling of smol_llama into a 8x220M MoE:

yml
base_model: BEE-spoke-data/smol_llama-220M-GQA
gate_mode: random
dtype: bfloat16
experts:
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
  - source_model: BEE-spoke-data/smol_llama-220M-GQA
# and then train the sucker!

Shove some Mistral models in a clown car:

yml
base_model: NousResearch/Hermes-2-Pro-Mistral-7B
gate_mode: hidden
dtype: bfloat16
experts:
  - source_model: NousResearch/Hermes-2-Pro-Mistral-7B
    positive_prompts:
      - "<|im_start|>user\nHello, who are you?<|im_end|>"
      - "<|im_start|>user\nI need help with"
  - source_model: BioMistral/BioMistral-7B-DARE
    positive_prompts:
      - "As a doctor of medicine,"
  - source_model: PocketDoc/Dans-AdventurousWinds-7b
    positive_prompts:
      - "[Genres: Science Fiction]\n[Tags: humor, old school, sci fi]"
      - "> get ye flask"
      - "[Mode: Interactive Storyteller]"
  - source_model: VAGOsolutions/SauerkrautLM-7b-HerO
    positive_prompts:
      - "<|im_start|>user\nWie geht es dir?<|im_end|>"
      - "Das ist ein Satz auf Deutsch."

FAQ

What does the "Your model has duplicated tensors but the --clone-tensors flag is not set" warning mean?

Answer from Charles O. Goddard (cg123)
(also see this GitHub issue):

This is completely benign. This happens when a single tensor from a model is used in multiple places, like when doing sparse upcycling with the moe script or doing passthrough merges that repeat layers. Having --clone-tensors set can use slightly more memory, but having it unset will slow down saving and introduce small memory usage spikes in cases where this warning occurs. It's honestly a small enough difference that the warning could be removed entirely.

Multimerge

mergekit-multi: Multi-Stage Model Merging

What is mergekit-multi?

mergekit-multi is a command-line tool for executing complex model merging workflows with multiple interdependent stages. It allows you to:

  1. Chain multiple merge operations together
  2. Use outputs from previous merges as inputs to subsequent ones
  3. Automatically handle dependencies between merge steps
  4. Cache intermediate results for faster re-runs

Usage

Basic command structure:

bash
mergekit-multi <config.yaml> \
  --intermediate-dir ./intermediates \
  ([--out-path ./final-merge] | if config has unnamed merge) \
  [options]

Configuration File Format

Create a YAML file with multiple merge configurations separated by ---. Each should contain:

  • name: Unique identifier for intermediate merges (except final merge)
  • Standard mergekit configuration parameters

Example with Final Merge (multimerge.yaml):

yaml
name: first-merge
merge_method: linear
models:
  - model: mistralai/Mistral-7B-v0.1
  - model: BioMistral/BioMistral-7B
parameters:
  weight: 0.5
---
name: second-merge
merge_method: slerp
base_model: first-merge  # Reference previous merge
models:
  - model: NousResearch/Hermes-2-Pro-Mistral-7B
parameters:
  t: 0.5
---
# Final merge (no name)
merge_method: dare_ties
base_model: mistralai/Mistral-7B-v0.1
models:
  - model: second-merge
    parameters:
      density: 0.6
      weight: 0.5
  - model: teknium/OpenHermes-2.5-Mistral-7B
    parameters:
      density: 0.8
      weight: 0.5

Example with All Named Merges:

yaml
name: first-merge
merge_method: task_arithmetic
...
---
name: second-merge
merge_method: slerp
...
---
name: third-merge
merge_method: linear
...

Key Options

  • --intermediate-dir: Directory to store partial merge results
  • --out-path: Output path for final merge (only applies when one merge has no name)
  • --lazy/--no-lazy: Don't rerun existing intermediate merges (default: true)
  • Standard mergekit options apply (e.g., --cuda, --out-shard-size, --multi-gpu)

How It Works

When you run mergekit-multi, it topologically sorts your merge configurations to determine the correct order of execution. The merges are then processed sequentially, using outputs from previous steps as inputs for subsequent ones as needed.

All intermediate merges are saved in your specified --intermediate-dir using their configured names. By default, the tool will skip any merge operations that already have existing output files. To force re-execution of all merges, use the --no-lazy flag.

Tokensurgeon

mergekit-tokensurgeon

mergekit-tokensurgeon is a command line utility for "transplanting" tokenizers between models. It reconstructs embeddings for a donor tokenizer inside the base model's embedding space so that the resulting model can operate with the donor vocabulary and ID mapping.

The default approach uses Orthogonal Matching Pursuit (OMP) to approximate unseen token embeddings as sparse combinations of tokens shared between the two vocabularies. This provides a training-free way to align tokenizers with minimal loss in downstream performance. The method is described in detail in the paper Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit.

Other approximation strategies are also implemented (e.g. common-vocabulary interpolation, subword based methods, PCA and more). You can control which technique is used via the --approximation-method option described below.

Usage

bash
mergekit-tokensurgeon \
  path/to/base_model \
  path/to/donor_model \
  ./output_model \
  [options]

This command creates a new model at ./output_model whose tokenizer matches the donor model. The main embeddings and language modeling head are updated so existing weights remain aligned with the new vocabulary.

Key Options

  • --k: Sparsity level (e.g., for omp, stb, mp_rope) or number of neighbors (e.g., for common_interpolation). (default: 64).
  • --approximation-method: One of omp (default), common_interpolation, subword, mean, zero, randn, john_hewitt, landmark_pca, stb, or mp_rope.
  • --weight-scheme: Weighting scheme for common interpolation (distance_proportional, barycentric, least_squares).
  • --subword-method: How to combine subword pieces when using the subword method (mean, sum, weighted_mean, first_last).
  • --prefix-match / --byte-match: Reuse existing embeddings that share a prefix or byte representation with donor tokens.
  • --magikarp: Filter out poorly trained tokens using the Magikarp heuristic before approximation.
  • Standard mergekit options such as --device and --trust-remote-code are also accepted.

Run mergekit-tokensurgeon --help for the full list of arguments.

Approximation Methods

mergekit-tokensurgeon implements a number of strategies for generating embeddings for tokens that do not exist in the base model. The method is selected with --approximation-method:

  • omp – Orthogonal Matching Pursuit (default). Approximates each missing token as a sparse linear combination of up to --k shared tokens.
  • common_interpolation – Finds the nearest overlapping tokens and interpolates between them using one of several weighting schemes (controlled with --weight-scheme).
  • subword – Breaks the token into pieces using the base tokenizer and combines their embeddings according to --subword-method.
  • landmark_pca – Builds a linear map between the donor and base embedding spaces from the shared tokens using PCA and applies it to the new tokens.
  • stb – Forms sparse (approximately) orthogonal token bases for both models and transfers coefficients between them.
  • mp_rope – Matching pursuit that accounts for rotary position embeddings to better align positional structure.
  • john_hewitt – Samples from the distribution of the base embeddings to generate new vectors. See John Hewitt's page for details.
  • mean, zero, randn – Simpler heuristics that fill new tokens with the base embedding mean, all zeros, or Gaussian noise respectively.

Practical Tips

  • For most models we recommend --approximation-method omp --k 64, which balances quality and compute cost.
  • The script operates entirely offline; no additional training is required.
  • Large discrepancies in numerical tokenization schemes can degrade math-heavy tasks. If both tokenizers split numbers in a similar way the transplanted model usually retains its arithmetic ability.

Further Reading

The theoretical motivation, implementation details and empirical evaluation of the OMP approach are presented in the accompanying paper. The mergekit-tokensurgeon tool exposes these techniques for practical use in merging workflows.

CONTRIBUTING

Contributing to MergeKit

Thank you for your interest in contributing to MergeKit! We welcome all contributions, from bug fixes to new features. This document outlines the guidelines and process for contributing, including how to set up your development environment, make changes, and submit pull requests.

Reporting Issues

If you encounter any bugs or have feature requests, please report them on the GitHub Issues page.
Before submitting a new issue, please search existing issues to see if your problem or suggestion has already been reported.

When reporting an issue, please try to provide as much detail as possible, including:

  • A clear and descriptive title
  • Steps to reproduce the issue
  • Expected behavior
  • Actual behavior
  • Merge configuration (if applicable)
  • Any relevant logs or error messages
  • Your environment (OS, Python version, MergeKit version)

Contributor License Agreement (CLA)

Before your contributions can be accepted, you must sign our Contributor License Agreement (CLA).
This is a one-time process, automated via the CLA Assistant Lite bot on GitHub. When you submit your first pull request, the bot will comment on it with instructions to sign the CLA electronically.

Development Environment Setup

  1. Fork the Repository: Click the "Fork" button on the top right of this page to create a copy of the repository under your GitHub account.
  2. Clone Your Fork:
bash
git clone https://github.com/YOUR-USERNAME/mergekit.git
    cd mergekit

Replace YOUR-USERNAME with your GitHub username.

  1. Configure Upstream Remote:

It's helpful to have a remote pointing to the original repository to fetch updates:

bash
git remote add upstream https://github.com/arcee-ai/mergekit.git
  1. Set Up a Virtual Environment (Recommended):

We recommend using uv for managing virtual environments. For installation, see the uv documentation.

bash
uv venv .venv
    source .venv/bin/activate # or on Windows: .venv\Scripts\activate

Alternatively, you can use venv or virtualenv or conda or whatever. I'm not the boss of you. For example, using venv:

bash
python3 -m venv .venv
    source .venv/bin/activate
  1. Install Dependencies:

Install MergeKit in editable mode, along with development and testing dependencies:

bash
uv pip install -e ".[test,dev]"
    # Or, using pip directly:
    # pip install -e ".[test,dev]"
  1. Install Pre-commit Hooks:

MergeKit uses pre-commit for automated code formatting.
To install the pre-commit hooks, run:

bash
pre-commit install

Contribution Workflow

  1. Sync Your main Branch:

Before creating a new branch, ensure your local main branch is synchronized with the upstream main branch:

bash
git checkout main
    git fetch upstream
    git merge upstream/main  # or rebase if you're feeling fancy
    git push origin main     # Optional: Keeps your fork's main branch updated
  1. Create a Branch: Create a new branch from your up-to-date main branch for your changes.
bash
git checkout -b my-feature-branch # e.g., fix/readme-typo or feat/new-merge-algorithm
  1. Make Your Changes:

Write your code, add tests, and update documentation as necessary. Commit however you like - pull requests will always be squashed before merging, so don't worry too much about keeping your commit history clean.

  1. Run Pre-commit Hooks:

The pre-commit hooks installed earlier will run automatically when you git commit. You can also run them manually on all files:

bash
pre-commit run --all-files

This will format your code and check for any linting issues. Make sure all checks pass before proceeding, as pull requests cannot be merged if these fail.

  1. Run Tests:

Run the test suite to ensure everything is working as expected and no regressions have been introduced.

bash
pytest tests/

All tests must pass before your contribution can be merged.

  1. Push Your Changes:

Push your changes to your forked repository.

bash
git push origin my-feature-branch

If you get an error because the remote branch doesn't exist yet, you might need:

bash
git push --set-upstream origin my-feature-branch

Submitting a Pull Request (PR)

  1. Open a Pull Request:

Navigate to the original MergeKit repository on GitHub. GitHub usually detects recently pushed branches from forks and will display a prompt to create a PR. If not, click the "New pull request" button.

  1. Target Branch:

Ensure your PR targets the main branch of the arcee-ai/mergekit repository. The "base" repository should be arcee-ai/mergekit and base branch main. The "head" repository should be your fork and the compare branch should be my-feature-branch.

  1. PR Title and Description:

* Provide a clear and concise title for your PR.
* In the description, explain the "what" and "why" of your changes.
* Link any related issues by typing # followed by the issue number (e.g., Closes #123). This helps automatically close the issue when the PR is merged.
* If your PR is a work in progress, consider creating it as a draft. You can mark it as ready for review later.

Pull Request Review Process

  1. Automated Checks: CI (Continuous Integration) checks will run automatically (e.g., tests, linters). Ensure these pass. If they fail, please investigate the logs, fix the issues, and push the changes to your branch.
  2. Maintainer Review: One or more maintainers will review your PR. They may ask for changes, offer suggestions, or request clarifications.
  3. Address Feedback: Please address any comments and push updates to your branch. The PR will update automatically with your new commits.
  4. Approval and Merge: Once the PR is approved and all checks pass, a maintainer will merge your contribution. Congratulations and thank you for your contribution!

Additional Resources

README

mergekit

[](https://www.gnu.org/licenses/lgpl-3.0)
[](https://github.com/arcee-ai/mergekit/actions/workflows/pre-commit.yml)
[](https://discord.gg/arceeai)

mergekit is a toolkit for merging pre-trained language models. mergekit uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as little as 8 GB of VRAM. Many merging algorithms are supported, with more coming as they catch my attention.

Contents

- Contributing
- Community Tools

- Parameter Specification
- Tokenizer Configuration
- Chat Template Configuration
- Examples

Why Merge Models?

Model merging is a powerful technique that allows combining the strengths of different models without the computational overhead of ensembling or the need for additional training. By operating directly in the weight space of models, merging can:

  • Combine multiple specialized models into a single versatile model
  • Transfer capabilities between models without access to training data
  • Find optimal trade-offs between different model behaviors
  • Improve performance while maintaining inference costs
  • Create new capabilities through creative model combinations

Unlike traditional ensembling which requires running multiple models, merged models maintain the same inference cost as a single model while often achieving comparable or superior performance.

Features

Key features of mergekit include:

Installation

sh
git clone https://github.com/arcee-ai/mergekit.git
cd mergekit

pip install -e .  # install the package and make scripts available

If the above fails with the error of:

text
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode:
(A "pyproject.toml" file was found, but editable mode currently requires a setuptools-based build.)

You may need to upgrade pip to > 21.3 with the command python3 -m pip install --upgrade pip.

Community & Support

Contributing

We welcome contributions to mergekit! If you have ideas for new merge methods, features, or other improvements, please check out our contributing guide for details on how to get started.

Community Tools

  • FrankensteinAI: For those who prefer a browser-based experience without local setup or hardware wrangling, the team at FrankensteinAI has built a hosted platform powered by mergekit. Also features a community gallery and leaderboard for sharing and comparing merged models.

Usage

The script mergekit-yaml is the main entry point for mergekit. It takes a YAML configuration file and an output path, like so:

sh
mergekit-yaml path/to/your/config.yml ./output-model-directory [--cuda] [--lazy-unpickle] [--allow-crimes] [... other options]

This will run the merge and write your merged model to ./output-model-directory.

For more information on the arguments accepted by mergekit-yaml run the command mergekit-yaml --help.

Uploading to Huggingface

When you have a merged model you're happy with, you may want to share it on the Hugging Face Hub. mergekit generates a README.md for your merge with some basic information for a model card. You can edit it to include more details about your merge, like giving it a good name or explaining what it's good at; rewrite it entirely; or use the generated README.md as-is. It is also possible to edit your README.md online once it has been uploaded to the Hub.

Once you're happy with your model card and merged model, you can upload it to the Hugging Face Hub using the huggingface_hub Python library.

sh
# log in to huggingface with an access token (must have write permission)
huggingface-cli login
# upload your model
huggingface-cli upload your_hf_username/my-cool-model ./output-model-directory .

The documentation for huggingface_hub goes into more detail about other options for uploading.

Merge Configuration

Merge configurations are YAML documents specifying the operations to perform in order to produce your merged model.
Below are the primary elements of a configuration file:

  • merge_method: Specifies the method to use for merging models. See Merge Methods for a list.
  • slices: Defines slices of layers from different models to be used. This field is mutually exclusive with models.
  • models: Defines entire models to be used for merging. This field is mutually exclusive with slices.
  • base_model: Specifies the base model used in some merging methods.
  • parameters: Holds various parameters such as weights and densities, which can also be specified at different levels of the configuration.
  • dtype: Specifies the data type used for the merging operation.
  • tokenizer or tokenizer_source: Determines how to construct a tokenizer for the merged model.
  • chat_template: Specifies a chat template for the merged model.

Parameter Specification

Parameters are flexible and can be set with varying precedence. They can be specified conditionally using tensor name filters, which allows finer control such as differentiating between attention heads and fully connected layers.

Parameters can be specified as:

  • Scalars: Single floating-point values.
  • Gradients: List of floating-point values, specifying an interpolated gradient.

The parameters can be set at different levels, with decreasing precedence as follows:

  1. slices.*.sources.parameters - applying to a specific input slice
  2. slices.*.parameters - applying to a specific output slice
  3. models.*.parameters or input_model_parameters - applying to any tensors coming from specific input models
  4. parameters - catchall

Tokenizer Configuration

The tokenizer behavior can be configured in two ways: using the new tokenizer field (recommended) or the legacy tokenizer_source field (maintained for backward compatibility). These fields are mutually exclusive - you should use one or the other, not both.

Modern Configuration (tokenizer)

The tokenizer field provides fine-grained control over vocabulary and embeddings:

yaml
tokenizer:
  source: "union"  # or "base" or a specific model path
  tokens:          # Optional: configure specific tokens
    <token_name>:
      source: ...  # Specify embedding source
      force: false # Optional: force this embedding for all models
  pad_to_multiple_of: null  # Optional: pad vocabulary size

##### Tokenizer Source

The source field determines the vocabulary of the output model:

  • union: Combine vocabularies from all input models (default)
  • base: Use vocabulary from the base model
  • "path/to/model": Use vocabulary from a specific model

##### Token Embedding Handling

When a tokenizer is configured, each input model's embedding matrix is adjusted to match the output vocabulary before being passed to the merge method. For tokens a model already has, its own embedding is used. For tokens a model is missing, a fallback embedding is assigned using these rules:

  • If the base model has the token, use the base model's embedding
  • If only one model has the token, use that model's embedding
  • Otherwise, use an average of all available embeddings

The merge method then combines these per-model embeddings (original and filled-in) to produce the final output. This means the final embedding for a token present in multiple models is determined by your merge method (SLERP, linear, TIES, etc.), not simply taken from one model.

You can override these defaults for specific tokens. Any tokens listed here that don't already exist in the output vocabulary will be added automatically, making this useful for introducing new special tokens.

yaml
tokenizer:
  source: union
  tokens:
    # Use embedding from a specific model
    <|im_start|>:
      source: "path/to/chatml/model"

    # Force a specific embedding for all models
    <|special|>:
      source: "path/to/model"
      force: true

    # Map a token to another model's token embedding
    <|renamed_token|>:
      source:
        kind: "model_token"
        model: "path/to/model"
        token: "<|original_token|>"  # or use token_id: 1234

    # Use a zero embedding
    <|unused|>:
      source:
        kind: "zero"

##### Practical Example

Here's how you might preserve both Llama 3 Instruct and ChatML prompt formats when merging models:

yaml
tokenizer:
  source: union
  tokens:
    # ChatML tokens
    <|im_start|>:
      source: "chatml_model"
    <|im_end|>:
      source: "chatml_model"

    # Llama 3 tokens - force original embeddings
    <|start_header_id|>:
      source: "llama3_model"
      force: true
    <|end_header_id|>:
      source: "llama3_model"
      force: true
    <|eot_id|>:
      source: "llama3_model"
      force: true

Legacy Configuration (tokenizer_source)

For backward compatibility, the tokenizer_source field is still supported:

yaml
tokenizer_source: "union"  # or "base" or a model path

This provides basic tokenizer selection but lacks the fine-grained control of the modern tokenizer field.

Chat Template Configuration

The optional chat_template field allows overriding the chat template used for the merged model.

yaml
chat_template: "auto"  # or a template name or Jinja2 template

Options include:

  • "auto": Automatically select the most common template among input models
  • Built-in templates: "alpaca", "chatml", "llama3", "mistral", "exaone"
  • A Jinja2 template string for custom formatting

Examples

Several examples of merge configurations are available in examples/.

Merge Methods

mergekit offers many methods for merging models, each with its own strengths and weaknesses. Choosing the right method depends on your specific goals, the relationship between the models you're merging, and the desired characteristics of the final model.

For detailed explanations, parameter descriptions, and use cases for each method, please see our Merge Method Guide.

Method Overview

Method (value)Core Idea# ModelsBase ModelKey Strengths / Use Cases
Linear (linear)

Simple weighted average of model parameters. | ≥2 | - | Averaging similar checkpoints, model soups. |
| SLERP (slerp) | Spherical linear interpolation between two models. | 2 | ✓ | Smoothly transitioning between two models. |
| NuSLERP (nuslerp) | Enhanced SLERP with flexible weighting. | 2 | * | More intuitive SLERP; task vector SLERP. |
| Multi-SLERP (multislerp) | Barycentric SLERP for multiple models. | ≥2 | * | Spherical interpolation for >2 models. |
| Karcher Mean (karcher) | Riemannian barycenter of model parameters. | ≥2 | - | Geometrically sound averaging on manifolds. |
| Task Arithmetic (task_arithmetic) | Linearly combine "task vectors" (differences from a base). | ≥2 | ✓ | Transferring/combining fine-tuned skills. |
| TIES (ties) | Task arithmetic + sparsification & sign consensus. | ≥2 | ✓ | Merging many models, reducing interference. |
| DARE (dare_linear, dare_ties) | Task arithmetic + random pruning & rescaling. | ≥2 | ✓ | Robust skill retention, similar to TIES. |
| DELLA (della, della_linear) | Task arithmetic + adaptive magnitude-based pruning. | ≥2 | ✓ | Prioritizing important changes, reducing interference. |
| Model Breadcrumbs (breadcrumbs, breadcrumbs_ties) | Task arithmetic + outlier removal (small & large diffs). | ≥2 | ✓ | Refining task vectors by removing extreme changes. |
| SCE (sce) | Task arithmetic + adaptive matrix-level weighting based on variance. | ≥2 | ✓ | Dynamically weighting models based on parameter variance. |
| Model Stock (model_stock) | Geometric weight calculation for linear interpolation. | ≥3 | ✓ | Finding good linear interpolation weights for many checkpoints. |
| Nearswap (nearswap) | Interpolate where parameters are similar. | 2 | ✓ | Selective merging based on parameter similarity. |
| Arcee Fusion (arcee_fusion) | Dynamic thresholding for fusing important changes. | 2 | ✓ | Identifying and merging salient features. |
| Passthrough (passthrough) | Directly copies tensors from a single input model. | 1 | - | Frankenmerging, layer stacking, model surgery. |

Key for Base Model Column:

  • ✓: Required - One of the input models must be designated as the base_model.
  • : Optional - One of the input models can* be designated as the base_model.
  • -: Not Applicable - base_model has no effect on this method.

LoRA Extraction

Mergekit allows extracting PEFT-compatible low-rank approximations of finetuned models.

Usage

sh
mergekit-extract-lora --model finetuned_model_id_or_path --base-model base_model_id_or_path --out-path output_path [--no-lazy-unpickle] [--cuda] [--max-rank=desired_rank] [--sv-epsilon=tol]

Mixture of Experts Merging

The mergekit-moe script supports merging multiple dense models into a mixture of experts, either for direct use or for further training. For more details see the mergekit-moe documentation.

Evolutionary Merge Methods

See docs/evolve.md for details.

Multi-Stage Merging (`mergekit-multi`)

mergekit-multi enables the execution of complex, multi-stage model merging workflows. You can define multiple merge configurations in a single YAML file, where later merges can use the outputs of earlier ones as inputs. This is useful for building up sophisticated models through a series of targeted merges.

See the mergekit-multi documentation for usage details and examples.

Raw PyTorch Model Merging (`mergekit-pytorch`)

For merging arbitrary PyTorch models (not necessarily Hugging Face Transformers), mergekit-pytorch provides a way to apply mergekit's algorithms directly to .pt or .safetensors checkpoints. The configuration is similar to the YAML format used in mergekit-yaml, but does not support layer slicing or tokenizer configuration.

Usage

sh
mergekit-pytorch path/to/your/raw_config.yml ./output_pytorch_model_directory [options]

Use mergekit-pytorch --help for detailed options.

Tokenizer Transplantation (`mergekit-tokensurgeon`)

mergekit-tokensurgeon is a specialized tool for transplanting tokenizers between models, allowing you to align the vocabulary of one model with another. This is particularly useful for cheaply producing draft models for speculative decoding or for cross-tokenizer knowledge distillation. See the documentation for more details and how to use it.

Citation

If you find mergekit useful in your research, please consider citing the paper:

text
/* Detailed source-code truncated for AI context efficiency. */

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#llama Browse llama catalog → #llm Browse llm catalog → #model-merging Browse model-merging catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →