### Create A Merge Method # Extending MergeKit with Custom Merge Methods ## Overview MergeKit offers two different paths for implementing custom merge methods: | | Decorator API | Class-based API | | ---------------------- | --------------------- | ---------------------------------------------- | | **Complexity** | Simple function-based | Full class implementation | | **Abstraction Level** | Higher-level | Lower-level | | **Parameter Handling** | Automatic validation | Manual configuration | | **Execution Flow** | Single function | Arbitrary computation graph | | **Best For** | Most merge methods | Complex multi-stage, multi-input strategies | Either approach benefits from MergeKit's underlying task system for resource management and execution control. The question of which to use largely depends on the complexity of the merge operation and the level of control needed. **Note on Parameter Configuration:** MergeKit uses a hierarchical YAML-based configuration system. Parameters for your custom merge methods (both scalar and per-model) can be defined at various levels (e.g., globally, per-model, per-slice). The values your merge function or task receives are resolved by MergeKit based on this hierarchy and context. For full details on configuration structure and parameter precedence, please refer to the [Merge Configuration](../README.md#merge-configuration) section of the README. ### Core Task System Features MergeKit's computational graph infrastructure provides sophisticated resource management that all merge methods inherit: - **Smart Memory Management** - Automatic return value lifecycle tracking - Early value eviction when no longer needed - Optimized shard loading based on task groups - **Device Management** - Automatic tensor movement between compute and storage devices - Support for both CPU and GPU execution - **Task Scheduling** - Tasks grouped by tensor shard to minimize memory usage - Loads deferred until last possible moment (via priority system) - Execution ordered to optimize shard residency ### Decorator API Best for straightforward merge operations that can be expressed as a single tensor transformation. Features: - Parameter validation, type checking, and value resolution - Configuration schema generation - Simplified base model handling - Default GPU acceleration opt-in ### Class-based API Choose when you need: - Multi-stage merge operations - Custom computation graphs - Direct access to weight metadata - Complex parameter types - Fine-grained control over execution ## Decorator API Implementation ### Basic Workflow 1. Define a type-annotated Python function with your merge logic 2. Add the `@merge_method` decorator with configuration 3. Ensure the module containing your function is imported - For MergeKit to discover your decorated merge method, the Python module containing it must be imported during MergeKit's initialization. Add an import statement for your module in `mergekit/merge_methods/__init__.py`. Once the module is imported, the `@merge_method` decorator handles the registration of the method with MergeKit. ### Example: Weighted Average ```python from mergekit.merge_methods.easy_define import merge_method from typing import List import torch @merge_method( name="weighted_average", pretty_name="Weighted Average", # Optional: human-readable name reference_url="https://example.com/docs", # Optional: documentation or paper link ) def average_merge( tensors: List[torch.Tensor], # Required: input tensors weight: List[float], # Vector parameter (one float per model) normalize: bool = True, # Scalar parameter with default ) -> torch.Tensor: if normalize: total = sum(weight) weight = [w / total for w in weight] return sum(t * w for t, w in zip(tensors, weight)) ``` This enables configurations like: ```yaml merge_method: weighted_average models: - model: model1 parameters: weight: 0.3 - model: model2 parameters: weight: 0.7 parameters: # Global parameters normalize: true ``` ### Parameter Types and Handling The decorator supports three parameter categories: 1. **Scalar Parameters** - Types: `bool`, `float`, or `int` - Single value for all models - Without defaults they become required parameters - Example: `normalize: bool = True` 2. **Vector Parameters** - Types: `List[float]` or `List[int]` only - Configured per-model - Default values must be single numbers, not lists, as they are broadcasted - Example: `weights: List[float]` 3. **Base Model Integration** The `tensors: List[torch.Tensor]` argument and an optional `base_tensor` argument in your function signature interact with the `base_model` specified in the YAML configuration as follows: - **If your function includes a `base_tensor` parameter (e.g., `base_tensor: torch.Tensor` or `base_tensor: Optional[torch.Tensor]`):** - The `base_tensor` argument will receive the tensor from the `base_model` specified in the YAML. If annotated as `Optional` and no `base_model` is configured, it will be `None`. - The `tensors: List[torch.Tensor]` argument will *only* contain tensors from the models specified under the `models:` key in the YAML, in order. It will *not* include the base model's tensor. - **If your function does *not* include a `base_tensor` parameter:** - If a `base_model` is specified in the YAML, its tensor will be the *first element* in the `tensors: List[torch.Tensor]` list (i.e., `tensors[0]`). - Subsequent elements (`tensors[1:]`) will correspond to the models listed under the `models:` key in the YAML, in order. - If no `base_model` is specified, the `tensors` list will directly correspond to the models listed under the `models:` key. 4. **Special Auto-Populated Parameters** Certain parameter names in your function signature have special meaning and are auto-populated by MergeKit if present. You do not configure these directly in the YAML `parameters` sections for your method; MergeKit provides them. - `output_weight: WeightInfo`: If your function accepts an argument named `output_weight` annotated with `WeightInfo`, MergeKit will pass metadata about the specific weight tensor being computed. - `base_model: ModelReference` (or `Optional[ModelReference]`): If your function accepts `base_model` annotated with `ModelReference`, MergeKit will pass a reference to the base model if one is used in the configuration for this merge operation. ## Class-based API Implementation For complex merges requiring granular control, implement `MergeMethod` and `Task` classes: ### Example Implementation ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Task Scheduling System The class-based API provides fine-grained control over execution: - **Priority Control**: Override `priority()` to influence execution order within groups - **Task Grouping**: Use `group_label()` to batch similar operations - **Resource Management**: - Automatic tensor lifecycle tracking - Memory optimization via early tensor eviction - Smart device placement for computation vs storage - **Computation Graph**: Build complex flows by connecting multiple tasks ### Implementation Requirements 1. Task Class: - Must implement `execute()` with proper type annotations - Must implement `arguments()` to declare dependencies - Optionally override `priority()`, `group_label()`, `uses_accelerator()` 2. Method Class: - Must implement core methods: `name()`, `make_task()` - Optional methods: `pretty_name()`, `reference_url()` - Define parameters via `parameters()` and `tensor_parameters()` ### Registration Add class-based methods to `STATIC_MERGE_METHODS` in `mergekit/merge_methods/registry.py`: ```python from mergekit.merge_methods.my_module import CustomMerge STATIC_MERGE_METHODS: List[MergeMethod] = [ CustomMerge(), # other methods... ] ``` ## Reference Implementations 1. **Linear Merge** (`mergekit.merge_methods.linear`): - Basic weighted averaging - Good example of class-based implementation 2. **Multi-SLERP** (`mergekit.merge_methods.multislerp`): - Hypersphere interpolation - Complex decorator usage example 3. **Task Arithmetic** (`mergekit.merge_methods.task_arithmetic`): - Advanced graph-based implementation - TIES/Magnitude pruning example --- ### Evolve # mergekit-evolve `mergekit-evolve` is a script that uses an evolutionary algorithm (CMA-ES) to optimize the parameters of a merge against model metrics. This is inspired by SakanaAI's [Evolutionary Optimization of Model Merging Recipes](https://arxiv.org/abs/2403.13187), in particular their parameter-space approach. `mergekit-evolve` uses EleutherAI's [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to define and evaluate the scoring function. The script is set up to be run either single-node or on a Ray cluster and has a few different strategies for scheduling operations depending on your particular configuration of compute. ## Installation Install `mergekit` with the `evolve` (and optionally `vllm`) features: ```sh git clone https://github.com/arcee-ai/mergekit.git cd mergekit pip install -e .[evolve,vllm] ``` If you had a perfectly good pytorch environment going and installing an older version of vLLM downgraded it and broke flash attention, run the following commands to fix it: ```sh pip uninstall flash-attn pip cache purge pip install flash-attn ``` ## Configuration `mergekit-evolve` takes in a YAML configuration file that defines how the merge is parameterized and what metrics to optimize. The general syntax is as follows: ```yml genome: models: - model_1 - model_2 ... - model_n merge_method: dare_ties base_model: base_model_if_needed tokenizer_source: null # optional layer_granularity: 8 # optional: normalize: false allow_negative_weights: false smooth: false filters: ... tasks: - name: lm_eval_task_name weight: 1.0 # optional metric: "acc,none" # defaults to acc,none - name: ... # as many as you want ``` ### Genome Definition The `genome` section of the configuration file defines the parameter space that `mergekit-evolve` will be optimizing in. #### `models` This should be a list of all of the models you want available to be merged. Depending on the merge method not all are guaranteed to be used in the final merge. #### `merge_method` Merge method to be used. Currently supported values are `linear`, `dare_ties`, `task_arithmetic`, `ties`, and `slerp`. #### `base_model` The base model for the merge, if applicable. #### `layer_granularity` A set of parameters will be introduced for each consecutive slice of `layer_granularity` layers. So for example, a 32-layer model like `mistralai/Mistral-7B-v0.1` with `layer_granularity: 8` will be divided into 4 groups of 8 layers with different merge parameters for each. The value specified here must be a divisor of the number of layers in your input models. Large values of `layer_granularity` will reduce the search space greatly, meaning you will get faster convergence at the cost of a potentially less good global solution. When not set, one set of parameters will be used for all layers. #### `normalize` Sets the `normalize` flag when merging. For methods like `linear`, `ties`, and `dare_ties` this constrains the search space to a set of definitely valid models. Similarly to `layer_granularity`, this can greatly speed up convergence at the cost of ruling out oddball solutions that might score better than more standard merges. #### `allow_negative_weights` Pretty self explanatory. When this flag is not set, the absolute value of weight parameters is used. Sensible search space reduction for `linear` and `slerp`. For task arithmetic based methods you probably want `allow_negative_weights: true`. #### `smooth` If set to `true`, then parameter values will be interpolated across layers instead of assigning a single, fixed value to each block. #### `filters` Accepts a list of filters, as in `mergekit-yaml`, by which to separate the parameters. So, for example, setting filters as below for a Llama-based merge: ```yaml filters: - self_attn - mlp ``` Will divide up the merge parameters into three groups - self attention parameters, MLP parameters, and a third for everything else. Separating the parameters out like this can be very beneficial when merging models trained on different prompt formats. It also makes your parameter space three times as big though! ### Task Definition To evaluate the produced merges you need to specify a list of tasks supported by the EleutherAI LM evaluation harness. This can be either [built in tasks](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks) (don't be naughty) or tasks you define yourself (see the [New Task Guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md) for how). If your task does not use `acc` as the metric then you must specify the correct metric name. Each task can also optionally have a weight associated. `mergekit-evolve` aims to maximize the score of the merge, so if you are using any tasks or metrics where a lower score is better (like perplexity) be sure to assign a negative weight to that task. ## Running `mergekit-evolve` ```sh mergekit-evolve [OPTIONS] --storage-path PATH GENOME_CONFIG_PATH ``` `mergekit-evolve` needs a storage path specified, where it will save the input models, merges to evaluate, and the config for the current best merge evaluated. If you are not using in-memory merging this can require a _lot_ of space - expect at least one fp16 model per GPU. Some important options: ### Scheduling Strategy (`--strategy`) There are three different strategies implemented for scheduling merging and evaluation jobs. #### `pool` Assigns an actor to each GPU in your cluster and guarantees merges and evaluations are performed on the same node. This is a safe default suitable for any configuration, local or distributed. #### `buffered` Maintains a buffer of tasks scheduled to ensure that there is always a model merging or ready to evaluate for each GPU. Allows for concurrent merging and evaluation of models on the same GPU if enough VRAM is available. Only suitable for a single-node setup or when `--storage-path` points to a fast shared filesystem. #### `serial` Uses Ray placement groups to ensure merges and their evaluations happen on the same node, but otherwise just lets Ray take the wheel. Maybe give a try if you're having trouble with the other two, otherwise probably don't use it. ### Evaluation LLM Backend By default `mergekit-evolve` will use the `hf` backend for `lm-eval`. To use vLLM instead, pass the `--vllm` flag. ### On-Disk vs. In-Memory By default `mergekit-evolve` will perform merges, write the result to disk, then start up an instance of lm-eval pointing at that path. This is a safe default and will generally always work but also causes a lot of GPU downtime and eats disk space. When using the `pool` scheduling strategy, you have the option to instead keep a model resident in memory and directly update its parameters instead of merging to disk. This is much faster and uses no additional disk space. However, it does involve mucking around in the internals of vLLM and the LM evaluation harness. So it might break at any moment! Choose wisely. Use `--in-memory` to enable this mode. ### Task search path If you're using custom task definitions (and you should be) then you can append to the search path using the `--task-search-path` option. This should point to the directory your custom task YAML is in (or a parent of that directory). Multiple paths can be included by repeating the option. ### Batch size Override the batch size used during merge evaluation. If using vLLM `auto` is recommended (default). ### CMA-ES options #### `--max-fevals` Maximum number of merges to evaluate. Note that the `cma` package is very loosey-goosey with this number and will happily go over by 50% depending on the size of each generation. Set to 100 by default. #### `--sigma0` Initial value of sigma for CMA-ES. No need to play with this unless you really know what you're doing. ### WandB logging `mergekit-evolve` supports logging metrics to Weights & Biases. Enable this functionality with the `--wandb` flag. Project and entity names can be overridden with the `--wandb-project` and `--wandb-entity` options. ### Example ```sh mergekit-evolve --strategy pool --wandb --wandb-project mergekit-evolve --wandb-entity arcee-ai --storage-path /path/to/mergekit-evolve/ ./config.yml ``` ## Output `mergekit-evolve` will write the merge configuration for the best merge found so far to the storage path with the filename `best_config.yaml`. If you're using WandB it will also log the config as an artifact. The script will keep running until a KeyboardInterrupt is received or `--max-fevals` is generously exceeded. ## Caveats `mergekit-evolve` is a work in progress and has probably not been tested on your specific configuration. Keep an eye on the output before leaving it running, and if you run in to any issues don't hesitate to file an issue! ## Acknowledgements Thanks to SakanaAI for the inspiration and the EleutherAI team for the LM evaluation harness. --- ### Merge Methods # Merge Method Guide ## Table of Contents - [Overview](#overview) - [Basic Merging Methods](#basic-merging-methods) - [Linear (`linear`)](#linear-linear) - [Spherical Interpolation Methods](#spherical-interpolation-methods) - [SLERP (`slerp`)](#slerp-slerp) - [NuSLERP (`nuslerp`)](#nuslerp-nuslerp) - [Multi-SLERP (`multislerp`)](#multi-slerp-multislerp) - [Karcher Mean (`karcher`)](#karcher-mean-karcher) - [Task Vector Methods](#task-vector-methods) - [Task Arithmetic (`task_arithmetic`)](#task-arithmetic-task_arithmetic) - [TIES-Merging (`ties`)](#ties-merging-ties) - [DARE (`dare_linear`, `dare_ties`)](#dare-dare_linear-dare_ties) - [DELLA (`della`, `della_linear`)](#della-della-della_linear) - [Model Breadcrumbs (`breadcrumbs`, `breadcrumbs_ties`)](#model-breadcrumbs-breadcrumbs-breadcrumbs_ties) - [SCE (`sce`)](#sce-sce) - [RAM (`ram`, `ramplus_tl`)](#ram-ram-ramplus_tl) - [Specialized Methods](#specialized-methods) - [Model Stock (`model_stock`)](#model-stock-model_stock) - [Nearswap (`nearswap`)](#nearswap-nearswap) - [Arcee Fusion (`arcee_fusion`)](#arcee-fusion-arcee_fusion) - [Passthrough (`passthrough`)](#passthrough-passthrough) - [Summary](#summary) - [Contributing](#contributing) ## Overview This guide provides detailed information about the various model merging algorithms available in `mergekit`. Each method has specific use cases, parameters, and applications for combining machine learning models. ## Basic Merging Methods ### Linear (`linear`) **Concept:** Computes a simple weighted average of the parameters from the input models. This is one of the most basic and widely used merging techniques. **Use Cases:** - Averaging multiple checkpoints of the same fine-tuning run ("model soups") - Combining models with very similar architectures and training data - Simple ensemble-like behavior in a single model **Inputs:** Takes 2 or more models. No `base_model` is typically used. **Key Parameters:** - `weight` (per-model): The contribution of each model to the average - `normalize` (global): If `true` (default), weights are normalized to sum to 1 **Reference:** [Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time](https://arxiv.org/abs/2203.05482) --- ## Spherical Interpolation Methods ### SLERP (`slerp`) **Concept:** Performs Spherical Linear Interpolation in the weight space between two models. This creates a path along a hypersphere, ensuring the interpolated model maintains a similar "norm" or "magnitude" to the original models. **Use Cases:** - Creating smooth transitions or intermediate points between two distinct models - Exploring the space between two models with potentially different capabilities **Inputs:** Requires exactly 2 models. One model must be specified as `base_model`. **Key Parameters:** - `t` (global): Interpolation factor. `t=0` yields the `base_model`, `t=1` yields the other model **Reference:** [Wikipedia: Slerp](https://en.wikipedia.org/wiki/Slerp) ### NuSLERP (`nuslerp`) **Concept:** An enhanced version of SLERP offering more flexible configuration and faster execution. It allows SLERP between two models directly. If a `base_model` is provided, NuSLERP calculates task vectors (the difference between each of the two main models and the base model) and then performs SLERP on these task vectors before adding the result back to the `base_model`. **Use Cases:** - Similar to SLERP, but with more control over weighting for the two primary models. - To replicate the behavior of the original slerp method (if `base_model` is *not* used, and weights are set to `1-t` for the first model and `t` for the second). - To perform SLERP on task vectors when a `base_model` is provided, allowing for interpolation of model changes *relative* to a common ancestor. **Inputs:** Requires exactly 2 models. A `base_model` can optionally be provided (it must be distinct from the two main models). **Key Parameters:** - `weight` (per-model): Relative weighting for each of the two main models. These are used to calculate the interpolation factor `t` (where `t = model2_weight / (model1_weight + model2_weight)`). - `nuslerp_flatten` (global): If `false`, performs row/column-wise interpolation. Default `true` - `nuslerp_row_wise` (global): If `true` (and `nuslerp_flatten` is `false`), SLERPs row vectors instead of column vectors. Default `false` ### Multi-SLERP (`multislerp`) **Concept:** Implements barycentric interpolation on a hypersphere for more than two models. It projects points onto a tangent space at their weighted Euclidean mean, performs interpolation, and projects back. **Use Cases:** - Creating a spherical average of multiple models - Finding a central point in the weight space of several related models **Inputs:** Takes 2 or more models. A `base_model` can optionally be provided to operate in task vector space. **Key Parameters:** - `weight` (per-model): Relative weighting for each model - `normalize_weights` (global): If `true` (default), weights are normalized - `eps` (global): Small constant for numerical stability. Default `1e-8` ### Karcher Mean (`karcher`) **Concept:** Computes the Karcher mean (also known as the Riemannian barycenter or Fréchet mean) of the input model parameters. This provides a geometrically sound way to average points on a manifold, which is suitable for model weights. **Use Cases:** - Finding a "central" or "average" model among a set of diverse models in a way that respects the geometry of the weight space - More robust averaging than simple linear averaging, especially for models far apart in weight space **Inputs:** Takes 2 or more models. No `base_model` is used. **Key Parameters:** - `max_iter` (global): Maximum iterations for the Karcher mean algorithm. Default `10` - `tol` (global): Convergence tolerance. Default `1e-5` **Reference:** [Functionality-Oriented LLM Merging on the Fisher-Rao Manifold](https://arxiv.org/abs/2603.04972) --- ## Task Vector Methods *The following methods build upon the concept of "task vectors," which represent the difference between a fine-tuned model and a base model.* ### Task Arithmetic (`task_arithmetic`) **Concept:** Computes "task vectors" for each model by subtracting a `base_model`. These task vectors are then combined as a weighted average and added back to the `base_model`. **Use Cases:** - Combining skills from multiple models fine-tuned from a common ancestor - Transferring specific capabilities (e.g., coding ability, instruction following) from one model to another - Steering style or behavior of a model by adding small task vectors from other models **Inputs:** Requires a `base_model` and one or more other models. **Key Parameters:** - `weight` (per-model): Weight for each model's task vector in the merge - `lambda` (global): Scaling factor applied to the summed task vectors before adding back to the base. Default `1.0` **Reference:** [Editing Models with Task Arithmetic](https://arxiv.org/abs/2212.04089) ### TIES-Merging (`ties`) **Concept:** Builds on Task Arithmetic by sparsifying task vectors and applying a sign consensus algorithm. This helps to resolve interference when merging multiple models and retain more of their individual strengths. **Use Cases:** - Merging a larger number of models effectively - Reducing parameter interference and negative synergy between merged models **Inputs:** Requires 2 or more models, plus one `base_model`. **Key Parameters:** - `weight` (per-model): Weight for each model's task vector - `density` (per-model): Fraction of weights to retain in each sparsified task vector - `lambda` (global): As in Task Arithmetic **Reference:** [TIES-Merging: Resolving Interference When Merging Models](https://arxiv.org/abs/2306.01708) ### DARE (`dare_linear`, `dare_ties`) **Concept:** Similar to TIES, DARE sparsifies task vectors to reduce interference. However, DARE uses random pruning with a novel rescaling technique to better match the performance of the original models. **Variants:** - `dare_linear`: DARE pruning without the TIES sign consensus - `dare_ties`: DARE pruning *with* the TIES sign consensus **Use Cases:** - Robustly combining multiple fine-tuned models, often yielding better performance than TIES in some scenarios **Inputs:** Requires 2 or more models, plus one `base_model`. **Key Parameters:** - `weight` (per-model): Weight for each model's task vector - `density` (per-model): Fraction of weights to retain after random pruning - `lambda` (global): As in Task Arithmetic - `rescale` (global, for `dare_linear`): If `true` (default), applies DARE's rescaling **Reference:** [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://arxiv.org/abs/2311.03099) ### DELLA (`della`, `della_linear`) **Concept:** Extends DARE by using adaptive pruning based on parameter magnitudes within each row of the delta parameters (task vectors). It calculates keep probabilities for each parameter: parameters with larger magnitudes within a row are assigned higher probabilities of being kept, while parameters with smaller magnitudes are assigned lower probabilities. These keep probabilities are scaled to range from `density - epsilon` (for the smallest magnitude element in a row) to `density + epsilon` (for the largest magnitude element in a row). This method aims to retain important changes while reducing interference, followed by DARE-like rescaling. **Variants:** - `della`: DELLA pruning with TIES sign consensus - `della_linear`: DELLA pruning without TIES sign consensus **Use Cases:** - Fine-grained control over pruning by prioritizing parameters with larger magnitude changes - Combining models where preserving the most significant changes is crucial **Inputs:** Requires 2 or more models, plus one `base_model`. **Key Parameters:** - `weight` (per-model): Weight for each model's task vector - `density` (per-model): Target fraction of weights to retain in differences from the base model - `epsilon` (per-model): Defines the half-width of the range for keep probabilities. Keep probabilities for parameters in a row will range from `density - epsilon` to `density + epsilon`, mapped from the smallest to largest magnitude parameters in that row, respectively. `epsilon` must be chosen such that `density - epsilon > 0` and `density + epsilon < 1`. - `lambda` (global): As in Task Arithmetic **Reference:** [DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling](https://arxiv.org/abs/2406.11617) ### Model Breadcrumbs (`breadcrumbs`, `breadcrumbs_ties`) **Concept:** An extension of task arithmetic designed to sparsify task vectors by pruning parameters with both the smallest and the largest absolute magnitudes (often considered outliers). This method operates in two main steps on the task vector (the difference between a fine-tuned model and the `base_model`): 1. First, a `gamma` fraction of the parameters with the *largest* absolute magnitudes are identified for removal. 2. Then, parameters with the *smallest* absolute magnitudes are identified for removal. The quantity of these smallest parameters to remove is determined such that the final `density` of parameters *retained* in the task vector is achieved, after accounting for the largest ones removed. The intention is to isolate and merge the "meaty," mid-range magnitude changes from the task vector, potentially filtering out noise (smallest changes) and overly dominant or conflicting large changes (largest changes). **Variants:** - `breadcrumbs`: Model Breadcrumbs pruning without TIES sign consensus - `breadcrumbs_ties`: Model Breadcrumbs pruning *with* TIES sign consensus **Use Cases:** - Merging models where extreme parameter changes might be detrimental or noisy - Refining task vectors by focusing on mid-range modifications, removing both the least significant and most extreme changes **Inputs:** Requires 2 or more models, plus one `base_model`. **Key Parameters:** - `weight` (per-model): Weight for each model's task vector. - `gamma` (per-model): The fraction of parameters with the *largest* absolute magnitudes in the task vector to be pruned (removed). For example, a `gamma` of `0.01` targets the removal of the top 1% of parameters with the highest absolute values. This parameter corresponds to `β` (beta) as described in the reference paper. - `density` (per-model): The final target fraction of parameters to *retain* in the task vector after both pruning steps (removal of largest `gamma` fraction and a corresponding fraction of smallest magnitude parameters). - The fraction of parameters with the *smallest* absolute magnitudes that will be pruned is calculated based on `density` and `gamma`. Specifically, it is `max(0, 1.0 - density - gamma)`. - **Example:** If `density: 0.9` and `gamma: 0.01`: - The top `0.01` (1%) largest magnitude parameters are removed. - The bottom `1.0 - 0.9 - 0.01 = 0.09` (9%) smallest magnitude parameters are also removed. - This results in `0.9` (90%) of the parameters being retained. - **Edge Case:** If `gamma` is set high enough such that `gamma >= 1.0 - density` (meaning `1.0 - density - gamma <= 0`), then the number of largest magnitude parameters actually pruned will be adjusted to `1.0 - density`, and no smallest magnitude parameters will be pruned (i.e., the fraction of smallest parameters pruned becomes 0). This ensures the `density` target is always respected and represents the fraction of parameters kept. - `lambda` (global): As in Task Arithmetic. **Reference:** [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://arxiv.org/abs/2312.06795) ### SCE (`sce`) **Concept:** The SCE (Select, Calculate, Erase) method performs adaptive matrix-level merging. It first computes task vectors (differences from the `base_model`). Then, it follows a three-step process for each parameter matrix (tensor): 1. **Select (Variance-Based Masking):** Optionally, parameter *positions* that show low variance across the different models' task vectors are identified and zeroed out. This is controlled by the `select_topk` parameter. If `select_topk < 1.0`, only the top `select_topk` fraction of parameter positions with the highest variance are kept active in the task vectors for subsequent steps. 2. **Calculate (Weighting):** Matrix-level merging coefficients (weights) are calculated for each model's task vector. These weights are derived from the mean of the squares of the elements within each task vector and are normalized across the models. 3. **Erase (Sign Consensus):** The sign-consensus algorithm from TIES is applied to the task vectors. Finally, the (variance-selected, calculated-weighted, and sign-agreed) task vectors are summed together, normalized by the sum of the effective applied weights at each position, and then added back to the `base_model`. **Use Cases:** - Dynamically weighting the contribution of different models at the matrix level based on parameter variance and calculated importance - Useful when some models contribute more significantly or consistently to certain parameter matrices than others - Merging models by focusing on high-variance, consistently signed changes **Inputs:** Requires 2 or more models, plus one `base_model`. **Key Parameters:** - `select_topk` (global): The fraction of parameter positions to retain based on their variance values across the different input models' task vectors. For each parameter position, variance is calculated across all task vectors. Only positions corresponding to the `select_topk` fraction with the highest variances are kept (i.e., their values in all task vectors are preserved for the next steps). Positions with lower variance are zeroed out in all task vectors. Set to 1.0 (default) to disable this variance-based selection step. This corresponds to `τ` (tau) in the reference paper. **Reference:** [FuseChat: Knowledge Fusion of Chat Models](https://arxiv.org/abs/2408.07990) ### RAM (`ram`, `ramplus_tl`) **Concept:** Reinforced Agent Merging (RAM) is a task vector method specifically designed for merging reinforcement learning (RL)-trained agents. The key insight is that on-policy RL induces task vectors that are highly sparse and heterogeneous, unlike the denser vectors from supervised fine-tuning. Traditional averaging methods dilute critical task-specific behaviors in this setting. RAM addresses this by classifying each parameter into three categories based on how many models modify it: - **Inactive**: No models modify this parameter (magnitude below `epsilon`) - **Unique**: Exactly one model modifies this parameter - **Shared**: Multiple models modify this parameter For inactive parameters, the base model value is preserved. Unique contributions are summed directly (preserving task-specific behaviors), while shared contributions are averaged (resolving conflicts). This selective merging strategy prevents the dilution of specialized agent behaviors. **Variants:** - `ram`: The base RAM method without rescaling. Unique parameters are preserved as-is, shared parameters are averaged. - `ramplus_tl`: RAM+ extends RAM by applying an adaptive rescaling factor `λ` to unique contributions, compensating for signal dilution caused by averaging shared parameters. The rescaling formula is: `λ = 1 + r · clip(ρ, 0, α)`, where `ρ` is the ratio of shared to unique parameter counts for each model within each tensor. `mergekit` implements a tensor-local variant, in which the overlap-unique ratio `ρ` is computed per-tensor rather than globally. **Use Cases:** - Scenarios where task vectors are sparse and heterogeneous (common in RL fine-tuning) - Merging multiple RL-trained agents from different tasks into a single generalist model **Inputs:** Requires a `base_model` and one or more other models. **Key Parameters:** - `epsilon` (global): Threshold for determining if a parameter is modified (values with absolute magnitude at or below `epsilon` are treated as inactive/unchanged). Default `1e-5` For `ramplus_tl` only: - `r` (global): Rescaling strength that controls the amplification of unique contributions. Higher values increase the reinforcement of unique parameters. Default `0.1` - `alpha` (global): Stability bound that clamps the overlap-unique ratio `ρ`, limiting the maximum rescaling factor. Default `0.2` **Reference:** [Reinforced Agent Merging](https://arxiv.org/abs/2601.13572) --- ## Specialized Methods ### Model Stock (`model_stock`) **Concept:** Uses geometric properties of fine-tuned models relative to a base_model to compute an optimized interpolation weight. It then performs a linear interpolation between the `base_model` and the average of the other input models using this computed weight. Specifically, task vectors (differences between other models and the `base_model`) are used to calculate pairwise cosine similarities. The average of these similarities informs the interpolation factor `t`. The final merged tensor is `t * average_of_other_models + (1 - t) * base_model`. **Use Cases:** - Finding effective weights for linearly combining multiple models where one model serves as a clear reference (`base_model`). - When a more principled, data-driven approach to linear interpolation between a base and a group of variants is desired over manual weight tuning. - Particularly strong for combining different training runs that were fine-tuned from the same `base_model` over the same or similar datasets. **Inputs:** Requires at least 3 models: one `base_model` and at least two other models. **Key Parameters:** - `filter_wise` (global): If `true`, weight calculation is per-row rather than per-tensor (not generally recommended). Default `false` **Reference:** [Model Stock: All we need is just a few fine-tuned models](https://arxiv.org/abs/2403.19522) ### Nearswap (`nearswap`) **Concept:** Interpolates the base model with parameters from a secondary model primarily where they are already similar. The interpolation strength towards the secondary model is inversely proportional to the absolute difference of their parameters, modulated by the `t` parameter. When the parameters are similar, the interpolation is stronger, and when they are different, it is weaker. **Use Cases:** - Selectively pulling in similar parameters from a secondary model while preserving different parameters from the base model - Fine-grained parameter-wise merging that respects the existing structure of the base model **Inputs:** Requires exactly 2 models. One model must be specified as `base_model`. **Key Parameters:** - `t` (global): Controls the interpolation strength. Higher values increase the influence of the secondary model for similar parameters **Algorithm:** For each parameter, computes `weight = (t / |base - secondary|).clamp(0, 1)`, then returns `weight * secondary + (1 - weight) * base` **Reference:** [QuartetAnemoi-70B-t0.0001 on Hugging Face](https://huggingface.co/alchemonaut/QuartetAnemoi-70B-t0.0001) ### Arcee Fusion (`arcee_fusion`) **Concept:** Merges two models by dynamically identifying and fusing important parameter changes. It calculates importance scores based on parameter differences and KL divergence, then uses a dynamic threshold to create a fusion mask. **Use Cases:** - Intelligently combining two models by prioritizing the most salient differences **Inputs:** Requires exactly 2 models. One model must be specified as `base_model`. **Key Parameters:** None beyond standard model selection **Reference:** [MergeKit v0.1 Release Blog](https://www.arcee.ai/blog/meet-mergekit-v0-1-arcee-fusion-expanded-model-support-multi-gpu-acceleration) ### Passthrough (`passthrough`) **Concept:** A no-op merge method that simply passes input tensors through unmodified from a single input model. **Use Cases:** - Layer-stacking or "Frankenmerging" where you assemble a model from specific unmodified layer ranges or individual tensors from one or more "donor" models - Useful as a building block in more complex `slices` configurations **Inputs:** Takes exactly 1 model. **Key Parameters:** - `scale` (per-model, optional): A scalar to multiply the tensor by. Useful for scaling specific layers, e.g., `{"filter": "down_proj", "value": 0.5}` --- ## Summary Merge methods serve different purposes and have different design goals. The choice of method depends on your specific use case, including the number of models, their relationships, and the desired characteristics of the final merged model. For beginners, starting with `linear`, `nuslerp`, or `task_arithmetic` can provide good results. For more advanced use cases, methods like `ties`, `dare_ties`, or `della` offer sophisticated ways to handle interference between multiple models while preserving their individual strengths. There is no "best" merge method; the right choice depends on your specific needs and the models you are working with, and selection is often more art than science. I encourage you to experiment with different methods and parameters vigorously to both find the best results for your use case and to learn more about how these methods work. Happy merging! ## Contributing If you have ideas for new merge methods or improvements to existing ones, we'd be glad to have you involved! Check out the [Contributing Guide](../CONTRIBUTING.md) and [Creating a Merge Method](create_a_merge_method.md) for more information on how to get started. --- ### Moe # mergekit-moe `mergekit-moe` is a script for combining Mistral or Llama models of the same size into Mixtral Mixture of Experts models. The script will combine the self-attention and layer normalization parameters from a "base" model with the MLP parameters from a set of "expert" models. If using the `hidden` or `cheap_embed` gate mode, the output model will be usable without any further training. If you are initializing a model to do further training on, such as for sparse upcycling, then use the `random` gate mode to get a model ready for training. ## Configuration `mergekit-moe` uses its own YML configuration syntax, which looks like so: ```yml base_model: path/to/self_attn_donor gate_mode: hidden # one of "hidden", "cheap_embed", or "random" dtype: bfloat16 # output dtype (float32, float16, or bfloat16) ## (optional) # experts_per_token: 2 experts: - source_model: expert_model_1 positive_prompts: - "This is a prompt that is demonstrative of what expert_model_1 excels at" ## (optional) # negative_prompts: # - "This is a prompt expert_model_1 should not be used for" - source_model: expert_model_2 # ... and so on ``` The script takes two arguments, an input config and an output path: `mergekit-moe ./config.yml ./my-clowncar-moe-12x180B` Currently the script can output models that use the Mixtral, Deepseek MoE, or Qwen MoE architectures. Some output architectures support a shared expert which will be activated for all tokens, which can be configured like this: ```yml base_model: path/to/self_attn_donor gate_mode: hidden # one of "hidden", "cheap_embed", or "random" dtype: bfloat16 # output dtype (float32, float16, or bfloat16) experts: ... shared_experts: - source_model: model_name positive_prompts: # required by Qwen MoE for "hidden" gate mode, otherwise not allowed - "blah blah" # (optional, but recommended:) residual_scale: 0.1 # downweight output from shared expert to prevent overcooking the model ``` Currently only up to one shared expert is supported. An appropriate architecture will be inferred based on the input models and presence or absence of shared experts in your configuration. Alternatively, you can explicitly specify an output architecture by setting the `architecture:` field in your config. For example: ```yml base_model: path/to/self_attn_donor architecture: qwen # ... and so on ``` ### Gate Modes There are three methods for populating the MoE gates implemented. #### "hidden" Uses the hidden state representations of the positive/negative prompts for MoE gate parameters. Best quality and most effective option; the default. Requires evaluating each prompt using the base model so you might not be able to use this on constrained hardware (depending on the model). You can use `--load-in-8bit` or `--load-in-4bit` to reduce VRAM usage. #### "cheap_embed" Uses only the raw token embedding of the prompts, using the same gate parameters for every layer. Distinctly less effective than "hidden". Can be run on much, much lower end hardware. #### "random" Randomly initializes the MoE gates. Good for if you are going to fine tune the model afterwards, or maybe if you want something a little unhinged? I won't judge. ## Example Configurations Sparse upcycling of smol_llama into a 8x220M MoE: ```yml base_model: BEE-spoke-data/smol_llama-220M-GQA gate_mode: random dtype: bfloat16 experts: - source_model: BEE-spoke-data/smol_llama-220M-GQA - source_model: BEE-spoke-data/smol_llama-220M-GQA - source_model: BEE-spoke-data/smol_llama-220M-GQA - source_model: BEE-spoke-data/smol_llama-220M-GQA - source_model: BEE-spoke-data/smol_llama-220M-GQA - source_model: BEE-spoke-data/smol_llama-220M-GQA - source_model: BEE-spoke-data/smol_llama-220M-GQA - source_model: BEE-spoke-data/smol_llama-220M-GQA # and then train the sucker! ``` Shove some Mistral models in a clown car: ```yml base_model: NousResearch/Hermes-2-Pro-Mistral-7B gate_mode: hidden dtype: bfloat16 experts: - source_model: NousResearch/Hermes-2-Pro-Mistral-7B positive_prompts: - "<|im_start|>user\nHello, who are you?<|im_end|>" - "<|im_start|>user\nI need help with" - source_model: BioMistral/BioMistral-7B-DARE positive_prompts: - "As a doctor of medicine," - source_model: PocketDoc/Dans-AdventurousWinds-7b positive_prompts: - "[Genres: Science Fiction]\n[Tags: humor, old school, sci fi]" - "> get ye flask" - "[Mode: Interactive Storyteller]" - source_model: VAGOsolutions/SauerkrautLM-7b-HerO positive_prompts: - "<|im_start|>user\nWie geht es dir?<|im_end|>" - "Das ist ein Satz auf Deutsch." ``` ## FAQ ### What does the "Your model has duplicated tensors but the --clone-tensors flag is not set" warning mean? Answer from [Charles O. Goddard (cg123)](https://github.com/cg123) (also see [this GitHub issue](https://github.com/arcee-ai/mergekit/issues/279#issuecomment-2081818104)): > This is completely benign. This happens when a single tensor from a model is used in multiple places, like when doing sparse upcycling with the moe script or doing passthrough merges that repeat layers. Having `--clone-tensors` set can use slightly more memory, but having it unset will slow down saving and introduce small memory usage spikes in cases where this warning occurs. It's honestly a small enough difference that the warning could be removed entirely. --- ### Multimerge # mergekit-multi: Multi-Stage Model Merging ## What is mergekit-multi? `mergekit-multi` is a command-line tool for executing complex model merging workflows with multiple interdependent stages. It allows you to: 1. Chain multiple merge operations together 2. Use outputs from previous merges as inputs to subsequent ones 3. Automatically handle dependencies between merge steps 4. Cache intermediate results for faster re-runs ## Usage Basic command structure: ```bash mergekit-multi \ --intermediate-dir ./intermediates \ ([--out-path ./final-merge] | if config has unnamed merge) \ [options] ``` ## Configuration File Format Create a YAML file with multiple merge configurations separated by `---`. Each should contain: - `name`: Unique identifier for intermediate merges (except final merge) - Standard mergekit configuration parameters Example with Final Merge (`multimerge.yaml`): ```yaml name: first-merge merge_method: linear models: - model: mistralai/Mistral-7B-v0.1 - model: BioMistral/BioMistral-7B parameters: weight: 0.5 --- name: second-merge merge_method: slerp base_model: first-merge # Reference previous merge models: - model: NousResearch/Hermes-2-Pro-Mistral-7B parameters: t: 0.5 --- # Final merge (no name) merge_method: dare_ties base_model: mistralai/Mistral-7B-v0.1 models: - model: second-merge parameters: density: 0.6 weight: 0.5 - model: teknium/OpenHermes-2.5-Mistral-7B parameters: density: 0.8 weight: 0.5 ``` ### Example with All Named Merges: ```yaml name: first-merge merge_method: task_arithmetic ... --- name: second-merge merge_method: slerp ... --- name: third-merge merge_method: linear ... ``` ## Key Options - `--intermediate-dir`: Directory to store partial merge results - `--out-path`: Output path for final merge (only applies when one merge has no `name`) - `--lazy/--no-lazy`: Don't rerun existing intermediate merges (default: true) - Standard mergekit options apply (e.g., `--cuda`, `--out-shard-size`, `--multi-gpu`) ## How It Works When you run `mergekit-multi`, it topologically sorts your merge configurations to determine the correct order of execution. The merges are then processed sequentially, using outputs from previous steps as inputs for subsequent ones as needed. All intermediate merges are saved in your specified `--intermediate-dir` using their configured names. By default, the tool will skip any merge operations that already have existing output files. To force re-execution of all merges, use the `--no-lazy` flag. --- ### Tokensurgeon # mergekit-tokensurgeon `mergekit-tokensurgeon` is a command line utility for "transplanting" tokenizers between models. It reconstructs embeddings for a donor tokenizer inside the base model's embedding space so that the resulting model can operate with the donor vocabulary and ID mapping. The default approach uses **Orthogonal Matching Pursuit (OMP)** to approximate unseen token embeddings as sparse combinations of tokens shared between the two vocabularies. This provides a training-free way to align tokenizers with minimal loss in downstream performance. The method is described in detail in the paper [*Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit*](https://arxiv.org/abs/2506.06607). Other approximation strategies are also implemented (e.g. common-vocabulary interpolation, subword based methods, PCA and more). You can control which technique is used via the `--approximation-method` option described below. ## Usage ```bash mergekit-tokensurgeon \ path/to/base_model \ path/to/donor_model \ ./output_model \ [options] ``` This command creates a new model at `./output_model` whose tokenizer matches the donor model. The main embeddings and language modeling head are updated so existing weights remain aligned with the new vocabulary. ### Key Options - `--k`: Sparsity level (e.g., for `omp`, `stb`, `mp_rope`) or number of neighbors (e.g., for `common_interpolation`). (default: 64). - `--approximation-method`: One of `omp` (default), `common_interpolation`, `subword`, `mean`, `zero`, `randn`, `john_hewitt`, `landmark_pca`, `stb`, or `mp_rope`. - `--weight-scheme`: Weighting scheme for common interpolation (`distance_proportional`, `barycentric`, `least_squares`). - `--subword-method`: How to combine subword pieces when using the `subword` method (`mean`, `sum`, `weighted_mean`, `first_last`). - `--prefix-match` / `--byte-match`: Reuse existing embeddings that share a prefix or byte representation with donor tokens. - `--magikarp`: Filter out poorly trained tokens using the Magikarp heuristic before approximation. - Standard mergekit options such as `--device` and `--trust-remote-code` are also accepted. Run `mergekit-tokensurgeon --help` for the full list of arguments. ## Approximation Methods `mergekit-tokensurgeon` implements a number of strategies for generating embeddings for tokens that do not exist in the base model. The method is selected with `--approximation-method`: - **omp** – Orthogonal Matching Pursuit (default). Approximates each missing token as a sparse linear combination of up to `--k` shared tokens. - **common_interpolation** – Finds the nearest overlapping tokens and interpolates between them using one of several weighting schemes (controlled with `--weight-scheme`). - **subword** – Breaks the token into pieces using the base tokenizer and combines their embeddings according to `--subword-method`. - **landmark_pca** – Builds a linear map between the donor and base embedding spaces from the shared tokens using PCA and applies it to the new tokens. - **stb** – Forms sparse (approximately) orthogonal token bases for both models and transfers coefficients between them. - **mp_rope** – Matching pursuit that accounts for rotary position embeddings to better align positional structure. - **john_hewitt** – Samples from the distribution of the base embeddings to generate new vectors. See [John Hewitt's page](https://www.cs.columbia.edu/~johnhew/vocab-expansion.html) for details. - **mean**, **zero**, **randn** – Simpler heuristics that fill new tokens with the base embedding mean, all zeros, or Gaussian noise respectively. ## Practical Tips - For most models we recommend `--approximation-method omp --k 64`, which balances quality and compute cost. - The script operates entirely offline; no additional training is required. - Large discrepancies in numerical tokenization schemes can degrade math-heavy tasks. If both tokenizers split numbers in a similar way the transplanted model usually retains its arithmetic ability. ## Further Reading The theoretical motivation, implementation details and empirical evaluation of the OMP approach are presented in the accompanying paper. The `mergekit-tokensurgeon` tool exposes these techniques for practical use in merging workflows. --- ### CONTRIBUTING # Contributing to MergeKit Thank you for your interest in contributing to MergeKit! We welcome all contributions, from bug fixes to new features. This document outlines the guidelines and process for contributing, including how to set up your development environment, make changes, and submit pull requests. ## Reporting Issues If you encounter any bugs or have feature requests, please report them on the [GitHub Issues page](https://github.com/arcee-ai/mergekit/issues). Before submitting a new issue, please search existing issues to see if your problem or suggestion has already been reported. When reporting an issue, please try to provide as much detail as possible, including: * A clear and descriptive title * Steps to reproduce the issue * Expected behavior * Actual behavior * Merge configuration (if applicable) * Any relevant logs or error messages * Your environment (OS, Python version, MergeKit version) ## Contributor License Agreement (CLA) Before your contributions can be accepted, you must sign our [Contributor License Agreement (CLA)](CLA.md). This is a one-time process, automated via the CLA Assistant Lite bot on GitHub. When you submit your first pull request, the bot will comment on it with instructions to sign the CLA electronically. ## Development Environment Setup 1. **Fork the Repository**: Click the "Fork" button on the top right of this page to create a copy of the repository under your GitHub account. 2. **Clone Your Fork**: ```bash git clone https://github.com/YOUR-USERNAME/mergekit.git cd mergekit ``` Replace `YOUR-USERNAME` with your GitHub username. 3. **Configure Upstream Remote**: It's helpful to have a remote pointing to the original repository to fetch updates: ```bash git remote add upstream https://github.com/arcee-ai/mergekit.git ``` 4. **Set Up a Virtual Environment** (Recommended): We recommend using [uv](https://github.com/astral-sh/uv) for managing virtual environments. For installation, see the [uv documentation](https://docs.astral.sh/uv/#installation). ```bash uv venv .venv source .venv/bin/activate # or on Windows: .venv\Scripts\activate ``` Alternatively, you can use `venv` or `virtualenv` or `conda` or whatever. I'm not the boss of you. For example, using `venv`: ```bash python3 -m venv .venv source .venv/bin/activate ``` 5. **Install Dependencies**: Install MergeKit in editable mode, along with development and testing dependencies: ```bash uv pip install -e ".[test,dev]" # Or, using pip directly: # pip install -e ".[test,dev]" ``` 6. **Install Pre-commit Hooks**: MergeKit uses [pre-commit](https://pre-commit.com/) for automated code formatting. To install the pre-commit hooks, run: ```bash pre-commit install ``` ## Contribution Workflow 1. **Sync Your `main` Branch**: Before creating a new branch, ensure your local `main` branch is synchronized with the upstream `main` branch: ```bash git checkout main git fetch upstream git merge upstream/main # or rebase if you're feeling fancy git push origin main # Optional: Keeps your fork's main branch updated ``` 2. **Create a Branch**: Create a new branch from your up-to-date `main` branch for your changes. ```bash git checkout -b my-feature-branch # e.g., fix/readme-typo or feat/new-merge-algorithm ``` 3. **Make Your Changes**: Write your code, add tests, and update documentation as necessary. Commit however you like - pull requests will always be squashed before merging, so don't worry too much about keeping your commit history clean. 4. **Run Pre-commit Hooks**: The pre-commit hooks installed earlier will run automatically when you `git commit`. You can also run them manually on all files: ```bash pre-commit run --all-files ``` This will format your code and check for any linting issues. Make sure all checks pass before proceeding, as pull requests cannot be merged if these fail. 5. **Run Tests**: Run the test suite to ensure everything is working as expected and no regressions have been introduced. ```bash pytest tests/ ``` All tests must pass before your contribution can be merged. 6. **Push Your Changes**: Push your changes to your forked repository. ```bash git push origin my-feature-branch ``` If you get an error because the remote branch doesn't exist yet, you might need: ```bash git push --set-upstream origin my-feature-branch ``` ## Submitting a Pull Request (PR) 1. **Open a Pull Request:** Navigate to the [original MergeKit repository](https://github.com/arcee-ai/mergekit) on GitHub. GitHub usually detects recently pushed branches from forks and will display a prompt to create a PR. If not, click the "New pull request" button. 2. **Target Branch:** Ensure your PR targets the `main` branch of the `arcee-ai/mergekit` repository. The "base" repository should be `arcee-ai/mergekit` and base branch `main`. The "head" repository should be your fork and the compare branch should be `my-feature-branch`. 3. **PR Title and Description:** * Provide a clear and concise title for your PR. * In the description, explain the "what" and "why" of your changes. * Link any related issues by typing `#` followed by the issue number (e.g., `Closes #123`). This helps automatically close the issue when the PR is merged. * If your PR is a work in progress, consider [creating it as a draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#draft-pull-requests). You can mark it as ready for review later. ## Pull Request Review Process 1. **Automated Checks:** CI (Continuous Integration) checks will run automatically (e.g., tests, linters). Ensure these pass. If they fail, please investigate the logs, fix the issues, and push the changes to your branch. 2. **Maintainer Review:** One or more maintainers will review your PR. They may ask for changes, offer suggestions, or request clarifications. 3. **Address Feedback:** Please address any comments and push updates to your branch. The PR will update automatically with your new commits. 4. **Approval and Merge:** Once the PR is approved and all checks pass, a maintainer will merge your contribution. Congratulations and thank you for your contribution! ## Additional Resources * [README](README.md): Overview of MergeKit and its features. * [Create a Merge Method](docs/create_a_merge_method.md): Guide for creating new merge methods. --- ### README # mergekit [](https://www.gnu.org/licenses/lgpl-3.0) [](https://github.com/arcee-ai/mergekit/actions/workflows/pre-commit.yml) [](https://discord.gg/arceeai) `mergekit` is a toolkit for merging pre-trained language models. `mergekit` uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as little as 8 GB of VRAM. Many merging algorithms are supported, with more coming as they catch my attention. ## Contents - [Why Merge Models?](#why-merge-models) - [Features](#features) - [Installation](#installation) - [Community & Support](#community--support) - [Contributing](#contributing) - [Community Tools](#community-tools) - [Usage](#usage) - [Merge Configuration](#merge-configuration) - [Parameter Specification](#parameter-specification) - [Tokenizer Configuration](#tokenizer-configuration) - [Chat Template Configuration](#chat-template-configuration) - [Examples](#examples) - [Merge Methods](#merge-methods) - [LoRA Extraction](#lora-extraction) - [Mixture of Experts Merging](#mixture-of-experts-merging) - [Evolutionary Merge Methods](#evolutionary-merge-methods) - [Multi-Stage Merging (`mergekit-multi`)](#multi-stage-merging-mergekit-multi) - [Raw PyTorch Model Merging (`mergekit-pytorch`)](#raw-pytorch-model-merging-mergekit-pytorch) - [Tokenizer Transplantation (`mergekit-tokensurgeon`)](#tokenizer-transplantation-mergekit-tokensurgeon) - [Citation](#citation) ## Why Merge Models? Model merging is a powerful technique that allows combining the strengths of different models without the computational overhead of ensembling or the need for additional training. By operating directly in the weight space of models, merging can: - Combine multiple specialized models into a single versatile model - Transfer capabilities between models without access to training data - Find optimal trade-offs between different model behaviors - Improve performance while maintaining inference costs - Create new capabilities through creative model combinations Unlike traditional ensembling which requires running multiple models, merged models maintain the same inference cost as a single model while often achieving comparable or superior performance. ## Features Key features of `mergekit` include: - Supports Llama, Mistral, GPT-NeoX, StableLM, and more - Many [merge methods](#merge-methods) - GPU or CPU execution - Lazy loading of tensors for low memory use - Interpolated gradients for parameter values (inspired by Gryphe's [BlockMerge_Gradient](https://github.com/Gryphe/BlockMerge_Gradient) script) - Piecewise assembly of language models from layers ("Frankenmerging") - [Mixture of Experts merging](#mixture-of-experts-merging) - [LORA extraction](#lora-extraction) - [Evolutionary merge methods](#evolutionary-merge-methods) - [Multi-stage merging](#multi-stage-merging-mergekit-multi) for complex workflows. - [Merging of raw PyTorch models (`mergekit-pytorch`)](#raw-pytorch-model-merging-mergekit-pytorch). ## Installation ```sh git clone https://github.com/arcee-ai/mergekit.git cd mergekit pip install -e . # install the package and make scripts available ``` If the above fails with the error of: ``` ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: (A "pyproject.toml" file was found, but editable mode currently requires a setuptools-based build.) ``` You may need to upgrade pip to > 21.3 with the command `python3 -m pip install --upgrade pip`. ## Community & Support - **Issues**: [GitHub Issues](https://github.com/arcee-ai/mergekit/issues) - **Discussions**: [Arcee Discord](https://discord.gg/arceeai) ### Contributing We welcome contributions to `mergekit`! If you have ideas for new merge methods, features, or other improvements, please check out our [contributing guide](CONTRIBUTING.md) for details on how to get started. ### Community Tools - **[FrankensteinAI](https://frankenstein-ai.com/)**: For those who prefer a browser-based experience without local setup or hardware wrangling, the team at FrankensteinAI has built a hosted platform powered by `mergekit`. Also features a community gallery and leaderboard for sharing and comparing merged models. ## Usage The script `mergekit-yaml` is the main entry point for `mergekit`. It takes a YAML configuration file and an output path, like so: ```sh mergekit-yaml path/to/your/config.yml ./output-model-directory [--cuda] [--lazy-unpickle] [--allow-crimes] [... other options] ``` This will run the merge and write your merged model to `./output-model-directory`. For more information on the arguments accepted by `mergekit-yaml` run the command `mergekit-yaml --help`. ### Uploading to Huggingface When you have a merged model you're happy with, you may want to share it on the Hugging Face Hub. `mergekit` generates a `README.md` for your merge with some basic information for a model card. You can edit it to include more details about your merge, like giving it a good name or explaining what it's good at; rewrite it entirely; or use the generated `README.md` as-is. It is also possible to edit your `README.md` online once it has been uploaded to the Hub. Once you're happy with your model card and merged model, you can upload it to the Hugging Face Hub using the [huggingface_hub](https://huggingface.co/docs/huggingface_hub/index) Python library. ```sh # log in to huggingface with an access token (must have write permission) huggingface-cli login # upload your model huggingface-cli upload your_hf_username/my-cool-model ./output-model-directory . ``` The [documentation](https://huggingface.co/docs/huggingface_hub/guides/cli#huggingface-cli-upload) for `huggingface_hub` goes into more detail about other options for uploading. ## Merge Configuration Merge configurations are YAML documents specifying the operations to perform in order to produce your merged model. Below are the primary elements of a configuration file: - `merge_method`: Specifies the method to use for merging models. See [Merge Methods](#merge-methods) for a list. - `slices`: Defines slices of layers from different models to be used. This field is mutually exclusive with `models`. - `models`: Defines entire models to be used for merging. This field is mutually exclusive with `slices`. - `base_model`: Specifies the base model used in some merging methods. - `parameters`: Holds various parameters such as weights and densities, which can also be specified at different levels of the configuration. - `dtype`: Specifies the data type used for the merging operation. - `tokenizer` or `tokenizer_source`: Determines how to construct a tokenizer for the merged model. - `chat_template`: Specifies a chat template for the merged model. ### Parameter Specification Parameters are flexible and can be set with varying precedence. They can be specified conditionally using tensor name filters, which allows finer control such as differentiating between attention heads and fully connected layers. Parameters can be specified as: - **Scalars**: Single floating-point values. - **Gradients**: List of floating-point values, specifying an interpolated gradient. The parameters can be set at different levels, with decreasing precedence as follows: 1. `slices.*.sources.parameters` - applying to a specific input slice 2. `slices.*.parameters` - applying to a specific output slice 3. `models.*.parameters` or `input_model_parameters` - applying to any tensors coming from specific input models 4. `parameters` - catchall ### Tokenizer Configuration The tokenizer behavior can be configured in two ways: using the new `tokenizer` field (recommended) or the legacy `tokenizer_source` field (maintained for backward compatibility). These fields are mutually exclusive - you should use one or the other, not both. #### Modern Configuration (tokenizer) The `tokenizer` field provides fine-grained control over vocabulary and embeddings: ```yaml tokenizer: source: "union" # or "base" or a specific model path tokens: # Optional: configure specific tokens : source: ... # Specify embedding source force: false # Optional: force this embedding for all models pad_to_multiple_of: null # Optional: pad vocabulary size ``` ##### Tokenizer Source The `source` field determines the vocabulary of the output model: - `union`: Combine vocabularies from all input models (default) - `base`: Use vocabulary from the base model - `"path/to/model"`: Use vocabulary from a specific model ##### Token Embedding Handling When a tokenizer is configured, each input model's embedding matrix is adjusted to match the output vocabulary before being passed to the merge method. For tokens a model already has, its own embedding is used. For tokens a model is *missing*, a fallback embedding is assigned using these rules: - If the base model has the token, use the base model's embedding - If only one model has the token, use that model's embedding - Otherwise, use an average of all available embeddings The merge method then combines these per-model embeddings (original and filled-in) to produce the final output. This means the final embedding for a token present in multiple models is determined by your merge method (SLERP, linear, TIES, etc.), not simply taken from one model. You can override these defaults for specific tokens. Any tokens listed here that don't already exist in the output vocabulary will be added automatically, making this useful for introducing new special tokens. ```yaml tokenizer: source: union tokens: # Use embedding from a specific model <|im_start|>: source: "path/to/chatml/model" # Force a specific embedding for all models <|special|>: source: "path/to/model" force: true # Map a token to another model's token embedding <|renamed_token|>: source: kind: "model_token" model: "path/to/model" token: "<|original_token|>" # or use token_id: 1234 # Use a zero embedding <|unused|>: source: kind: "zero" ``` ##### Practical Example Here's how you might preserve both Llama 3 Instruct and ChatML prompt formats when merging models: ```yaml tokenizer: source: union tokens: # ChatML tokens <|im_start|>: source: "chatml_model" <|im_end|>: source: "chatml_model" # Llama 3 tokens - force original embeddings <|start_header_id|>: source: "llama3_model" force: true <|end_header_id|>: source: "llama3_model" force: true <|eot_id|>: source: "llama3_model" force: true ``` #### Legacy Configuration (tokenizer_source) For backward compatibility, the `tokenizer_source` field is still supported: ```yaml tokenizer_source: "union" # or "base" or a model path ``` This provides basic tokenizer selection but lacks the fine-grained control of the modern `tokenizer` field. ### Chat Template Configuration The optional `chat_template` field allows overriding the chat template used for the merged model. ```yaml chat_template: "auto" # or a template name or Jinja2 template ``` Options include: - `"auto"`: Automatically select the most common template among input models - Built-in templates: `"alpaca"`, `"chatml"`, `"llama3"`, `"mistral"`, `"exaone"` - A Jinja2 template string for custom formatting ### Examples Several examples of merge configurations are available in [`examples/`](examples/). ## Merge Methods `mergekit` offers many methods for merging models, each with its own strengths and weaknesses. Choosing the right method depends on your specific goals, the relationship between the models you're merging, and the desired characteristics of the final model. For detailed explanations, parameter descriptions, and use cases for each method, please see our [**Merge Method Guide**](docs/merge_methods.md). ### Method Overview | Method (`value`) | Core Idea | # Models | Base Model | Key Strengths / Use Cases | |:----------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------|:--------:|:----:|:---------------------------------------------------------------| | [**Linear** (`linear`)](docs/merge_methods.md#linear-linear) | Simple weighted average of model parameters. | ≥2 | - | Averaging similar checkpoints, model soups. | | [**SLERP** (`slerp`)](docs/merge_methods.md#slerp-slerp) | Spherical linear interpolation between two models. | 2 | ✓ | Smoothly transitioning between two models. | | [**NuSLERP** (`nuslerp`)](docs/merge_methods.md#nuslerp-nuslerp) | Enhanced SLERP with flexible weighting. | 2 | * | More intuitive SLERP; task vector SLERP. | | [**Multi-SLERP** (`multislerp`)](docs/merge_methods.md#multi-slerp-multislerp) | Barycentric SLERP for multiple models. | ≥2 | * | Spherical interpolation for >2 models. | | [**Karcher Mean** (`karcher`)](docs/merge_methods.md#karcher-mean-karcher) | Riemannian barycenter of model parameters. | ≥2 | - | Geometrically sound averaging on manifolds. | | [**Task Arithmetic** (`task_arithmetic`)](docs/merge_methods.md#task-arithmetic-task_arithmetic) | Linearly combine "task vectors" (differences from a base). | ≥2 | ✓ | Transferring/combining fine-tuned skills. | | [**TIES** (`ties`)](docs/merge_methods.md#ties-merging-ties) | Task arithmetic + sparsification & sign consensus. | ≥2 | ✓ | Merging many models, reducing interference. | | [**DARE** (`dare_linear`, `dare_ties`)](docs/merge_methods.md#dare-dare_linear-dare_ties) | Task arithmetic + random pruning & rescaling. | ≥2 | ✓ | Robust skill retention, similar to TIES. | | [**DELLA** (`della`, `della_linear`)](docs/merge_methods.md#della-della-della_linear) | Task arithmetic + adaptive magnitude-based pruning. | ≥2 | ✓ | Prioritizing important changes, reducing interference. | | [**Model Breadcrumbs** (`breadcrumbs`, `breadcrumbs_ties`)](docs/merge_methods.md#model-breadcrumbs-breadcrumbs_ties) | Task arithmetic + outlier removal (small & large diffs). | ≥2 | ✓ | Refining task vectors by removing extreme changes. | | [**SCE** (`sce`)](docs/merge_methods.md#sce-sce) | Task arithmetic + adaptive matrix-level weighting based on variance. | ≥2 | ✓ | Dynamically weighting models based on parameter variance. | | [**Model Stock** (`model_stock`)](docs/merge_methods.md#model-stock-model_stock) | Geometric weight calculation for linear interpolation. | ≥3 | ✓ | Finding good linear interpolation weights for many checkpoints. | | [**Nearswap** (`nearswap`)](docs/merge_methods.md#nearswap-nearswap) | Interpolate where parameters are similar. | 2 | ✓ | Selective merging based on parameter similarity. | | [**Arcee Fusion** (`arcee_fusion`)](docs/merge_methods.md#arcee-fusion-arcee_fusion) | Dynamic thresholding for fusing important changes. | 2 | ✓ | Identifying and merging salient features. | | [**Passthrough** (`passthrough`)](docs/merge_methods.md#passthrough-passthrough) | Directly copies tensors from a single input model. | 1 | - | Frankenmerging, layer stacking, model surgery. | **Key for `Base Model` Column:** - ✓: **Required** - One of the input models *must* be designated as the `base_model`. - *: **Optional** - One of the input models *can* be designated as the `base_model`. - -: **Not Applicable** - `base_model` has no effect on this method. ## LoRA Extraction Mergekit allows extracting PEFT-compatible low-rank approximations of finetuned models. ### Usage ```sh mergekit-extract-lora --model finetuned_model_id_or_path --base-model base_model_id_or_path --out-path output_path [--no-lazy-unpickle] [--cuda] [--max-rank=desired_rank] [--sv-epsilon=tol] ``` ## Mixture of Experts Merging The `mergekit-moe` script supports merging multiple dense models into a mixture of experts, either for direct use or for further training. For more details see the [`mergekit-moe` documentation](docs/moe.md). ## Evolutionary Merge Methods See [`docs/evolve.md`](docs/evolve.md) for details. ## Multi-Stage Merging (`mergekit-multi`) `mergekit-multi` enables the execution of complex, multi-stage model merging workflows. You can define multiple merge configurations in a single YAML file, where later merges can use the outputs of earlier ones as inputs. This is useful for building up sophisticated models through a series of targeted merges. See the [`mergekit-multi` documentation](docs/multimerge.md) for usage details and examples. ## Raw PyTorch Model Merging (`mergekit-pytorch`) For merging arbitrary PyTorch models (not necessarily Hugging Face Transformers), `mergekit-pytorch` provides a way to apply mergekit's algorithms directly to `.pt` or `.safetensors` checkpoints. The configuration is similar to the YAML format used in `mergekit-yaml`, but does not support layer slicing or tokenizer configuration. ### Usage ```sh mergekit-pytorch path/to/your/raw_config.yml ./output_pytorch_model_directory [options] ``` Use `mergekit-pytorch --help` for detailed options. ## Tokenizer Transplantation (`mergekit-tokensurgeon`) `mergekit-tokensurgeon` is a specialized tool for transplanting tokenizers between models, allowing you to align the vocabulary of one model with another. This is particularly useful for cheaply producing draft models for speculative decoding or for cross-tokenizer knowledge distillation. See the [documentation](docs/tokensurgeon.md) for more details and how to use it. ## Citation If you find `mergekit` useful in your research, please consider citing the [paper](https://aclanthology.org/2024.emnlp-industry.36/): ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ---