{"owner":"huggingface","repo":"sentence-transformers","hasSkills":true,"totalSkillsCount":18,"totalTokensCount":23149,"categories":["anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":["skills/train-sentence-transformers/references/dataset_formats.md","skills/README.md","skills/train-sentence-transformers/references/evaluators_sentence_transformer.md","skills/train-sentence-transformers/references/evaluators_cross_encoder.md","skills/train-sentence-transformers/references/base_model_selection.md","skills/train-sentence-transformers/references/evaluators_multi_vector_encoder.md","skills/train-sentence-transformers/SKILL.md","skills/train-sentence-transformers/references/evaluators_sparse_encoder.md","skills/train-sentence-transformers/references/hardware_guide.md","skills/train-sentence-transformers/references/hf_jobs_execution.md","skills/train-sentence-transformers/references/losses_multi_vector_encoder.md","skills/train-sentence-transformers/references/losses_cross_encoder.md","skills/train-sentence-transformers/references/losses_sentence_transformer.md","skills/train-sentence-transformers/references/losses_sparse_encoder.md","skills/train-sentence-transformers/references/model_architectures.md","skills/train-sentence-transformers/references/prompts_and_instructions.md","skills/train-sentence-transformers/references/training_args.md","skills/train-sentence-transformers/references/troubleshooting.md"],"skills":{"skills/train-sentence-transformers/references/dataset_formats.md":"# Dataset Formats\n\nThis reference covers: how datasets map to losses, how to reshape data when it doesn't fit, and how to mine hard negatives.\n\n## The two rules\n\nFrom the sentence-transformers training overview:\n\n1. If the loss requires a label, the dataset must have a column named **`label`, `labels`, `score`, or `scores`**. Any column with one of these names is the label.\n2. All other columns are **inputs**. The loss defines how many input columns it expects. Column **names don't matter. Order does**.\n\nExample: `CoSENTLoss` expects 2 inputs + a float label. A dataset with columns `[\"premise\", \"hypothesis\", \"score\"]` works. A dataset with `[\"score\", \"premise\", \"hypothesis\"]` does not. Reorder first.\n\n## Per-loss data shapes\n\nThe per-type loss references (`losses_sentence_transformer.md`, `losses_cross_encoder.md`, `losses_sparse_encoder.md`) are the canonical mappings from data shape to loss. Cross-cutting recipe bits those tables don't show:\n\n- **`CosineSimilarityLoss`** wants `score` normalized to `[0, 1]`. `CoSENTLoss` / `AnglELoss` are pairwise-ranking and ignore absolute scale, so on `stsb` (raw 0-5) divide by 5 only when using cosine-similarity.\n- **`BatchAllTripletLoss` / `BatchHardTripletLoss` / `BatchSemiHardTripletLoss`** need `batch_sampler=BatchSamplers.GROUP_BY_LABEL` so multiple samples per label appear in the same batch.\n- **`MSELoss` (distillation)** label is the teacher's full embedding vector (a list of floats), not a scalar score.\n- **`MarginMSELoss` (distillation)** label is `teacher_score(q, pos) - teacher_score(q, neg)`, precomputed per row.\n- **N-tuple shape** for MNRL `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed) is produced by `mine_hard_negatives(..., output_format=\"n-tuple\")`. \"labeled-list\" output_format produces the CrossEncoder listwise shape.\n\n## Reshaping operations\n\nIf your data doesn't fit the loss's expected shape:\n\n### Reorder columns\n\n```python\n# Columns are [\"hypothesis\", \"premise\", \"score\"] but CoSENTLoss expects premise first.\ndataset = dataset.select_columns([\"premise\", \"hypothesis\", \"score\"])\n```\n\n### Rename label column\n\n```python\n# Your label is called \"relevance\" but ST wants \"label\".\ndataset = dataset.rename_column(\"relevance\", \"label\")\n```\n\n### Drop extra columns\n\n```python\n# ST will treat every non-label column as an input. Drop metadata.\ndataset = dataset.remove_columns([\"source_id\", \"created_at\", \"language\"])\n```\n\n### Convert dtypes\n\n```python\n# Label is str, need float for CoSENTLoss.\ndataset = dataset.map(lambda x: {\"label\": float(x[\"label\"])})\n```\n\n## Hard-negative mining\n\n`mine_hard_negatives` (in `sentence_transformers.util`) produces a training dataset with mined negatives using a retriever. Hard negatives are the single highest-leverage lever for retrieval-model quality.\n\n### Basic usage\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom sentence_transformers.util import mine_hard_negatives\n\nretriever = SentenceTransformer(\"sentence-transformers/all-MiniLM-L6-v2\")\n\nmined = mine_hard_negatives(\n dataset=train_pairs, # has (anchor, positive) or (q, a) columns\n model=retriever,\n num_negatives=5,\n range_min=0, range_max=100, # rank window to sample hard negatives from\n sampling_strategy=\"top\", # \"top\" = rank-1 hardest; \"random\" = random in window\n output_format=\"n-tuple\", # \"triplet\" | \"n-tuple\" | \"labeled-pair\" | \"labeled-list\"\n use_faiss=True,\n)\n```\n\n### Output formats\n\n- `\"triplet\"`: `(anchor, positive, negative)` triplets. One row per `(query, negative)` pair.\n- `\"n-tuple\"`: `(anchor, positive, negative_1, negative_2, ..., negative_N)` (1-indexed). One row per query.\n- `\"labeled-pair\"`: `(anchor, text, label)` with `label=1` for positives and `label=0` for negatives. Good for `BinaryCrossEntropyLoss`.\n- `\"labeled-list\"`: `(anchor, texts, labels)`. One row per query with a list of candidates. Good for listwise losses.\n\n### Filtering false negatives\n\nIf the retriever returns \"negatives\" that are actually relevant, they become false negatives and hurt training. Filter them:\n\n```python\nmined = mine_hard_negatives(\n dataset=train_pairs,\n model=retriever,\n cross_encoder=CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-6-v2\"), # score candidates\n num_negatives=5,\n max_score=0.9, # drop candidates scoring above 0.9\n relative_margin=0.05, # require neg_score <= pos_score - abs(pos_score) * 0.05\n absolute_margin=0.2, # require neg_score <= pos_score - 0.2\n output_format=\"n-tuple\",\n use_faiss=True,\n)\n```\n\nUse **either** `relative_margin` or `absolute_margin`, usually not both. `max_score` is independently useful as a hard ceiling.\n\n### CLI\n\n`scripts/mine_hard_negatives.py` is a CLI wrapper. See it for a ready-to-run command.\n\n## Choosing the right `range_min` / `range_max`\n\n`range_max=None` is the default. Pass an integer to cap how far down the ranked list to sample from.\n\n- `range_min=0`, `range_max=100`: sample from the top-100 retrieved. Good default.\n- `range_min=10`, `range_max=100`: skip the top-10 (often contains true positives). Safer if you lack a cross-encoder.\n- `range_min=0`, `range_max=1000`: wider net, more diverse negatives, slower.\n- `sampling_strategy=\"top\"`: always pick the rank-1 hardest. Maximum training signal per row.\n- `sampling_strategy=\"random\"`: pick randomly within the range. More robust if your retriever is itself noisy.\n\n## Quick Hub-side dataset checks\n\n`hf datasets sql \"SELECT * FROM 'hf://datasets//' LIMIT 5\"` streams rows via DuckDB without `load_dataset(...)`. This is the fastest way to confirm column names match your loss before a full validation run. `hf datasets info ` shows config / splits / size. `hf datasets card --text` renders the README.\n\n## Gotchas\n\n- **`remove_unused_columns=True` (default)**: the trainer drops columns that aren't passed to the model's forward. Usually fine, but if you rely on a custom collator that uses metadata columns, set `remove_unused_columns=False`.\n- **Floats stored as strings after CSV load**: `load_dataset(\"csv\", ...)` keeps columns as strings by default. Cast with `.map(lambda x: {\"label\": float(x[\"label\"])})`.\n- **Mined hard negatives with `include_positives=True`** include the positive as a negative in the output list, only useful when you're building an evaluator or want to measure rank of the positive. For training, leave it `False`.\n","skills/README.md":"# Sentence-Transformers Agent Skill\n\nThis is a tool-neutral [Agent Skill](https://agentskills.io) for training and fine-tuning `sentence-transformers` models. It covers model selection, hard-negative mining, loss / evaluator choice, training, evaluation, and Hub publishing across all four archetypes:\n\n| Archetype | What it does |\n|---|---|\n| `SentenceTransformer` (bi-encoder) | Fixed-dimension dense vectors for semantic similarity, retrieval, clustering, classification. |\n| `CrossEncoder` (reranker) | Pair scoring for two-stage retrieval / pairwise classification. |\n| `SparseEncoder` (SPLADE) | Sparse vectors over the vocabulary for learned-sparse retrieval. |\n| `MultiVectorEncoder` (ColBERT) | Per-token vectors scored with MaxSim for late-interaction retrieval, over texts or page images. |\n\nLoss / evaluator / example references inside the skill are split per archetype (`losses_sentence_transformer.md`, `losses_cross_encoder.md`, `losses_sparse_encoder.md`, `losses_multi_vector_encoder.md`, etc.). `SKILL.md` directs the agent to load only the ones matching the user's task.\n\n## Install\n\nYou don't need to clone this repo to use the skill. It's published to [`huggingface/skills`](https://github.com/huggingface/skills).\n\n### Codex, Cursor, Gemini CLI, OpenCode (Agent Skills standard)\n\n```bash\nhf skills add train-sentence-transformers # ./.agents/skills/\nhf skills add train-sentence-transformers --global # ~/.agents/skills/\n```\n\nAny agent that follows the [Agent Skills](https://agentskills.io) standard auto-discovers skills under `.agents/skills/`. `hf skills update` refreshes installed skills to the latest upstream commit.\n\n### Claude Code\n\nClaude Code reads from `.claude/skills/`, so the `--claude` flag also symlinks the install there:\n\n```bash\nhf skills add train-sentence-transformers --claude\n```\n\n## Using the skill\n\nOnce installed, just mention the task naturally, and the agent loads `SKILL.md` automatically based on its `description` frontmatter:\n\n> \"Train a multilingual sentence-transformer on my parallel corpus.\"\n> \"Fine-tune a cross-encoder reranker on `(question, answer)` pairs from my dataset, mine hard negatives, and push to my Hub repo.\"\n> \"Train a SPLADE model from `naver/splade-v3` on domain data and evaluate sparsity.\"\n> \"Fine-tune a ColBERT model from `lightonai/LateOn` on my retrieval data and evaluate it on NanoBEIR.\"\n\n## Local development\n\nIf you want to iterate on the skill from a local clone of this repo (so your agent picks up edits instantly without going through `hf skills add`), symlink the skill subdirectory into your agent's standard skill location. Symlinking the individual subdir (rather than the whole `skills/` directory) lets it coexist with any other skills you already have installed there.\n\n**macOS / Linux:**\n\n```bash\nmkdir -p .agents/skills\nln -sfn \"$(pwd)/skills/train-sentence-transformers\" .agents/skills/train-sentence-transformers\n```\n\n**Windows:**\n\n```cmd\nif not exist .agents\\skills mkdir .agents\\skills\nmklink /J \".agents\\skills\\train-sentence-transformers\" \"%cd%\\skills\\train-sentence-transformers\"\n```\n\nFor Claude Code, repeat with `.claude/skills/` as the target.\n\nAfter this, edits to `SKILL.md` / any reference / any script under `skills/train-sentence-transformers/` are immediately visible to your agent on the next skill invocation. If you only use one tool, drop the corresponding line for the other.\n\n## Maintenance\n\nThe skill is mirrored automatically to `huggingface/skills` on every release tag via [.github/workflows/sync-skills.yml](../.github/workflows/sync-skills.yml). The canonical source lives here in the `sentence-transformers` repo.\n","skills/train-sentence-transformers/references/evaluators_sentence_transformer.md":"# Evaluators (Bi-Encoder)\n\nAll bi-encoder evaluators live in `sentence_transformers.sentence_transformer.evaluation`.\n\n## Choosing the right evaluator\n\n| Task | Evaluator |\n|---|---|\n| Retrieval (nDCG, MRR, Recall), fast default | `NanoBEIREvaluator` |\n| Retrieval on your own corpus / qrels | `InformationRetrievalEvaluator` |\n| STS / continuous similarity | `EmbeddingSimilarityEvaluator` |\n| Binary classification | `BinaryClassificationEvaluator` |\n| Triplet accuracy | `TripletEvaluator` |\n| Reranking (from retrieval candidates) | `RerankingEvaluator` |\n| MSE vs. teacher (distillation) | `MSEEvaluator`, `MSEEvaluatorFromDataFrame` |\n| Paraphrase mining | `ParaphraseMiningEvaluator` |\n| Translation (cross-lingual alignment) | `TranslationEvaluator` |\n| Label accuracy (classification during training) | `LabelAccuracyEvaluator` |\n\nWrap multiple evaluators in `SequentialEvaluator` to track all of them together:\n\n```python\nfrom sentence_transformers.sentence_transformer.evaluation import SequentialEvaluator\nevaluator = SequentialEvaluator([evaluator1, evaluator2, evaluator3])\n```\n\n## The big three\n\n### `NanoBEIREvaluator` (retrieval)\n\nSmall, fast subset of BEIR. Typically runs in <1 minute on a mid-range GPU. Default choice for retrieval training.\n\n```python\nfrom sentence_transformers.sentence_transformer.evaluation import NanoBEIREvaluator\n\nevaluator = NanoBEIREvaluator(\n dataset_names=[\"msmarco\", \"nfcorpus\", \"nq\"], # default: all 13 NanoBEIR datasets\n batch_size=128,\n show_progress_bar=False,\n)\n```\n\n- Default dataset list covers 13 tasks. Pick a subset for speed during training.\n- Output key for `metric_for_best_model`: **`eval_NanoBEIR_mean_cosine_ndcg@10`** (bi-encoder default = cosine similarity).\n\n### `EmbeddingSimilarityEvaluator` (STS-style)\n\nComputes Pearson/Spearman correlation between model cosine similarities and gold labels.\n\n```python\nfrom sentence_transformers.sentence_transformer.evaluation import EmbeddingSimilarityEvaluator\nfrom sentence_transformers.util.similarity import SimilarityFunction\n\nevaluator = EmbeddingSimilarityEvaluator(\n sentences1=stsb[\"sentence1\"],\n sentences2=stsb[\"sentence2\"],\n scores=stsb[\"score\"],\n main_similarity=SimilarityFunction.COSINE,\n name=\"sts-dev\",\n)\n```\n\n- `main_similarity` can be `COSINE`, `DOT_PRODUCT`, `EUCLIDEAN`, `MANHATTAN`.\n- `name` is used in the output key: `eval_sts-dev_spearman_cosine`, `eval_sts-dev_pearson_cosine`, etc.\n\n### `InformationRetrievalEvaluator` (full retrieval)\n\nUse when you have your **own** corpus + queries + qrels (not one of the NanoBEIR tasks).\n\n```python\nfrom sentence_transformers.sentence_transformer.evaluation import InformationRetrievalEvaluator\n\nevaluator = InformationRetrievalEvaluator(\n queries={qid: query_text for qid, query_text in ...},\n corpus={doc_id: doc_text for doc_id, doc_text in ...},\n relevant_docs={qid: {doc_id, ...} for qid in ...}, # qid -> set of relevant doc_ids\n name=\"my-retrieval\",\n mrr_at_k=[10],\n ndcg_at_k=[10],\n accuracy_at_k=[1, 5, 10],\n precision_recall_at_k=[1, 5, 10],\n map_at_k=[100],\n show_progress_bar=False,\n batch_size=64,\n)\n```\n\nOutput keys: `eval_{name}_cosine_ndcg@10`, `eval_{name}_cosine_mrr@10`, etc.\n\nHeavy for large corpora: each eval encodes the full corpus. Don't run it every 100 steps. Use `NanoBEIREvaluator` for frequent evaluation during training and reserve full IR for milestones / post-training.\n\n## Other bi-encoder evaluators\n\n### `BinaryClassificationEvaluator`\n\nFor labeled pair classification (e.g. duplicate detection, entailment as binary). Reports accuracy, F1, precision/recall, AP. Supports all distance metrics. Finds the best threshold per metric.\n\n### `TripletEvaluator`\n\nFor `(anchor, positive, negative)` triplets. Reports the fraction of triplets where the positive is closer to the anchor than the negative.\n\n### `RerankingEvaluator`\n\nFor custom re-ranking datasets: you provide candidates per query, the evaluator computes MAP and MRR. Good for measuring retrieval-quality on a held-out set.\n\n### `MSEEvaluator` / `MSEEvaluatorFromDataFrame`\n\nFor distillation setups. Compares student embeddings against teacher embeddings (or teacher scores), reports MSE.\n\n### `ParaphraseMiningEvaluator`\n\nFor paraphrase-mining tasks. Given a corpus of labeled paraphrase pairs, computes mining quality (F1 at various thresholds).\n\n### `TranslationEvaluator`\n\nFor cross-lingual / `make_multilingual`-style alignment checking. Measures whether the student aligns sentences across languages.\n\n### `LabelAccuracyEvaluator`\n\nFor a `SoftmaxLoss`-trained classifier head. Reports accuracy on held-out data.\n\n## Writing `metric_for_best_model`\n\nPattern: `f\"eval_{evaluator.primary_metric}\"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:\n- `eval_NanoBEIR_mean_cosine_ndcg@10`: `NanoBEIREvaluator`\n- `eval_sts-dev_spearman_cosine`: `EmbeddingSimilarityEvaluator(name=\"sts-dev\")`\n- `eval_{name}_cosine_ndcg@10`: `InformationRetrievalEvaluator(name=...)`\n\n## Multi-dimensional evaluation (Matryoshka)\n\nFor Matryoshka-trained models, evaluate at each target dimension:\n\n```python\nper_dim_evaluators = [\n EmbeddingSimilarityEvaluator(\n sentences1=..., sentences2=..., scores=...,\n main_similarity=SimilarityFunction.COSINE,\n name=f\"sts-dev-{dim}\",\n truncate_dim=dim,\n ) for dim in [768, 512, 256, 128, 64]\n]\nevaluator = SequentialEvaluator(per_dim_evaluators, main_score_function=lambda scores: scores[0])\n```\n\nThe first evaluator's score drives `load_best_model_at_end`.\n\n## Gotchas\n\n- **Always run `evaluator(model)` once before training**: pre-training baseline. If the post-training delta is tiny, the loss/data/base is wrong.\n- Don't run `InformationRetrievalEvaluator` with a large corpus (>100k docs) at frequent `eval_steps`. Use `NanoBEIREvaluator` during training, reserve full IR for end-of-training.\n- `greater_is_better=True` is the default, right for nDCG / MRR / accuracy.\n","skills/train-sentence-transformers/references/evaluators_cross_encoder.md":"# Evaluators (Cross-Encoder)\n\nAll cross-encoder evaluators live in `sentence_transformers.cross_encoder.evaluation`.\n\n## Choosing the right evaluator\n\n| Task | Evaluator |\n|---|---|\n| Rerank retrieval results (nDCG@k on BM25 top-N), fast default | `CrossEncoderNanoBEIREvaluator` |\n| Rerank with custom candidates per query | `CrossEncoderRerankingEvaluator` |\n| Binary / multi-class pair classification | `CrossEncoderClassificationEvaluator` |\n| Continuous pair scoring (STS-style) | `CrossEncoderCorrelationEvaluator` |\n\nWrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`) to track them together:\n\n```python\nfrom sentence_transformers.base.evaluation import SequentialEvaluator\nevaluator = SequentialEvaluator([nano_beir_eval, custom_rerank_eval])\n```\n\n## The default: `CrossEncoderNanoBEIREvaluator`\n\nAnalog of `NanoBEIREvaluator` for rerankers. Takes BM25 top-100 for each NanoBEIR query and measures how well the cross-encoder re-ranks them.\n\n```python\nfrom sentence_transformers.cross_encoder.evaluation import CrossEncoderNanoBEIREvaluator\n\nevaluator = CrossEncoderNanoBEIREvaluator(\n dataset_names=[\"msmarco\", \"nfcorpus\", \"nq\"], # default: 11 of 13 NanoBEIR datasets (excludes \"arguana\", \"touche2020\")\n batch_size=64,\n rerank_k=100, # rerank the BM25 top-K\n)\n```\n\nOutput key for `metric_for_best_model`: **`eval_NanoBEIR_R100_mean_ndcg@10`**. The `R100` signals \"rerank top-100\". If you change `rerank_k`, the prefix changes (e.g. `R50`).\n\nEach individual dataset contributes `eval_Nano{DatasetName}_R100_ndcg@10` (e.g. `eval_NanoMSMARCO_R100_ndcg@10`) too.\n\n## Custom reranking with your own candidates\n\nUse when you have query + positive + distractor candidates that aren't part of NanoBEIR:\n\n```python\nfrom sentence_transformers.cross_encoder.evaluation import CrossEncoderRerankingEvaluator\n\nsamples = [\n {\"query\": \"...\", \"positive\": [\"the gold answer\"], \"documents\": [\"...\", \"...\", ...]}\n for ...\n]\n\nevaluator = CrossEncoderRerankingEvaluator(\n samples=samples,\n batch_size=64,\n name=\"my-rerank\",\n always_rerank_positives=False, # default is True; override to False for realistic eval\n)\n```\n\n- `always_rerank_positives=True` (the library default) forces the positive into the candidate pool even when the retriever missed it. The reranker is graded only on candidates it can actually score, so the metric reflects pure reranker quality.\n- `always_rerank_positives=False`: the positive is only reranked if it's already in `documents`. If the retriever missed it, the rank counts as N+1. This reflects end-to-end retriever+reranker quality. A positive the retriever missed is lost, regardless of reranker skill.\n\nOutput key: `eval_{name}_ndcg@10`, `eval_{name}_map`, `eval_{name}_mrr@10`.\n\n## Classification-style cross-encoders\n\n### `CrossEncoderClassificationEvaluator`\n\nWorks for both binary (`num_labels=1`) and multi-class (`num_labels>=2`) cross-encoders. Branches internally:\n- `num_labels=1`: binary mode. Sweeps thresholds to report accuracy, F1, precision, recall, and **average_precision** (primary).\n- `num_labels>=2`: multi-class mode (e.g. NLI: entailment / neutral / contradiction). Reports **f1_macro** (primary), f1_micro, f1_weighted, and per-class precision / recall.\n\n```python\nfrom sentence_transformers.cross_encoder.evaluation import CrossEncoderClassificationEvaluator\n\nevaluator = CrossEncoderClassificationEvaluator(\n sentence_pairs=[(premise, hypothesis), ...],\n labels=[0, 1, 2, ...],\n batch_size=64,\n name=\"nli-dev\",\n)\n```\n\nOutput keys (binary, `num_labels=1`): `eval_{name}_accuracy`, `eval_{name}_f1`, `eval_{name}_average_precision` (primary).\nOutput keys (multi-class, `num_labels>=2`): `eval_{name}_f1_macro` (primary), `eval_{name}_f1_micro`, `eval_{name}_f1_weighted`.\n\n### `CrossEncoderCorrelationEvaluator`\n\nFor continuous-score cross-encoders (like an STS cross-encoder outputting a similarity score). Reports Pearson/Spearman vs. gold scores.\n\n```python\nfrom sentence_transformers.cross_encoder.evaluation import CrossEncoderCorrelationEvaluator\n\nevaluator = CrossEncoderCorrelationEvaluator(\n sentence_pairs=[(a, b), ...],\n scores=[0.4, 0.8, ...],\n name=\"stsb-dev\",\n)\n```\n\nOutput keys: `eval_{name}_spearman`, `eval_{name}_pearson`.\n\n## Writing `metric_for_best_model`\n\nPattern: `f\"eval_{evaluator.primary_metric}\"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:\n- `eval_NanoBEIR_R100_mean_ndcg@10`: `CrossEncoderNanoBEIREvaluator` default\n- `eval_{name}_ndcg@10`: `CrossEncoderRerankingEvaluator`\n- `eval_{name}_average_precision`: `CrossEncoderClassificationEvaluator` (binary, `num_labels=1`)\n- `eval_{name}_f1_macro`: `CrossEncoderClassificationEvaluator` (multi-class, `num_labels>=2`)\n- `eval_{name}_spearman`: `CrossEncoderCorrelationEvaluator`\n\n## Gotchas\n\n- **Always run `evaluator(model)` once before training**: pre-training baseline. Tiny post-training delta means the loss/data/base is wrong.\n- `CrossEncoderClassificationEvaluator` accepts both `num_labels=1` (binary, primary `average_precision`) and `num_labels>=2` (multi-class, primary `f1_macro`). `CrossEncoderCorrelationEvaluator` requires `num_labels=1`.\n- The default `dataset_names=None` excludes `arguana` and `touche2020` (Argument-Retrieval task differs from the rest). Pass `dataset_names=list(DATASET_NAME_TO_HUMAN_READABLE)` from `sentence_transformers.cross_encoder.evaluation.nano_beir` to actually run all 13.\n- Subset NanoBEIR datasets during training (3-4) to keep eval cheap. Run the broader set post-training.\n","skills/train-sentence-transformers/references/base_model_selection.md":"# Base Model Selection\n\nLeaderboards rotate every few months. Don't trust any hardcoded \"best\" pick. Discover current options live. Run **both** sort orders since most-downloaded surfaces proven options and trending surfaces recent SOTA that may not have download volume yet.\n\n## Discovery commands\n\n**[BI]**:\n```bash\nhf models list --filter sentence-transformers --sort downloads --limit 20\nhf models list --filter sentence-transformers --sort trending --limit 20\n```\n\n**[CE]**:\n```bash\nhf models list --filter sentence-transformers --filter text-ranking --sort downloads --limit 20\nhf models list --filter sentence-transformers --filter text-ranking --sort trending --limit 20\n```\n\n**[SPARSE]**:\n```bash\nhf models list --filter sentence-transformers --filter sparse-encoder --sort downloads --limit 20\nhf models list --filter sentence-transformers --filter sparse-encoder --sort trending --limit 20\n```\n\nOptional language narrowing (any type): add `--filter `. Not all multilingual models tag each language, so missing matches doesn't mean the model can't handle that language. Re-run without the filter to compare.\n\n```bash\nhf models card --text # confirm dimensions, max_seq_length, license, languages\n```\n\nCross-check the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard) (pick the relevant tab) before committing to a multi-hour run.\n\n## [BI] Bi-Encoder\n\nContinue from an existing retriever beats fresh-start + 100k-500k pairs. Common namespaces as of 2026-Q2 (verify against discovery commands, since the field rotates):\n\n- **English encoder retrievers**: `sentence-transformers/all-*` (MiniLM-L6-v2, mpnet-base-v2 still the most-downloaded models on the Hub), `BAAI/bge-*-en-v1.5`, `nomic-ai/nomic-embed-text-v1.5`, `mixedbread-ai/mxbai-embed-large-v1`, `Alibaba-NLP/gte-*`, `Snowflake/snowflake-arctic-embed-*`, `jinaai/jina-embeddings-v5-text-small` / `-nano`, `microsoft/harrier-oss-v1-270m` / `-0.6b`.\n- **Multilingual encoder retrievers**: `sentence-transformers/paraphrase-multilingual-*`, `intfloat/multilingual-e5-*`, `ibm-granite/granite-embedding-*-multilingual-r2`, `google/embeddinggemma-300m`, `voyageai/voyage-4-nano`.\n- **Long documents (8k+)**: `nomic-ai/modernbert-embed-*`, `answerdotai/ModernBERT-large`.\n- **Decoder LLM retrievers** (multilingual, **need last-token pooling**): `Qwen/Qwen3-Embedding-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Embedding-*` (multimodal), `codefuse-ai/F2LLM-v2-*`.\n- **Fresh-start English** (≥500k pairs + domain-fit reason): `microsoft/mpnet-base`, `answerdotai/ModernBERT-base`, `google-bert/bert-base-uncased`, `jhu-clsp/ettin-encoder-*` (17m / 32m / 68m / 150m / 400m / 1b, paired ModernBERT encoder family).\n- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (MLM-only, needs contrastive training), `microsoft/mdeberta-v3-base`, `jhu-clsp/mmBERT-base` / `-small`.\n- **CPU / small footprint** (`StaticEmbedding`): `StaticEmbedding(tokenizer, embedding_dim=...)`. **Model size = `vocab_size × dim × 4 bytes`**. Pick a small-vocab tokenizer or you get a giant model: 30k-vocab `bert-base-uncased` × 128 dim ≈ 15 MB, **250k-vocab `paraphrase-multilingual-MiniLM-L12-v2` × 256 dim ≈ 256 MB**. Random init needs 1M+ pairs. Warm-start (`StaticEmbedding.from_distillation(...)`) helps under ~100k pairs.\n\nArchitecture variants (encoder / decoder / static / Router), pooling rules, and decoder-vs-encoder setup paths: `model_architectures.md`.\n\n**ModernBERT-family bases default to `max_seq_length=8192`.** That allocates activation memory for 8192-token sequences regardless of your data length and silently drives Windows VRAM into \"shared memory\" spillover. After loading any ModernBERT / mmBERT / Ettin / gte-modernbert / nomic-modernbert base, **explicitly set `model.max_seq_length = 256` (or 512 for documents)** unless you actually need long context.\n\n## [CE] Cross-Encoder\n\nContinue from an existing reranker beats fresh-start + 100k-500k pairs in most domains. Default to this unless you have a strong reason otherwise. Common namespaces as of 2026-Q2:\n\n- **English encoder rerankers**: `cross-encoder/ms-marco-*`, `BAAI/bge-reranker-*`, `mixedbread-ai/mxbai-rerank-*-v1` / `-v2`, `Alibaba-NLP/gte-reranker-modernbert-*`, `ibm-granite/granite-embedding-reranker-english-*`.\n- **Multilingual encoder rerankers**: `cross-encoder/mmarco-*`, `BAAI/bge-reranker-v2-m3`, `Alibaba-NLP/gte-multilingual-reranker-*`, `ibm-granite/granite-embedding-reranker-multilingual-*`.\n- **Decoder LLM rerankers** (multilingual, `num_labels=1` last-token-style scoring): `Qwen/Qwen3-Reranker-*` (0.6B / 4B / 8B), `Qwen/Qwen3-VL-Reranker-*` (multimodal).\n- **Fresh-start**: `microsoft/MiniLM-L12-H384-uncased`, `answerdotai/ModernBERT-base` / `-large`, `jhu-clsp/ettin-encoder-*`, `FacebookAI/xlm-roberta-base` (multilingual), `microsoft/mdeberta-v3-base` (multilingual), `jhu-clsp/mmBERT-base` / `-small` (multilingual). Pass `num_labels >= 2` for classification cross-encoders.\n\nEncoder-only bases are still the latency-efficient default (bidirectional attention is well-suited to the reranking use case at small parameter counts), but decoder LLM rerankers are now competitive at the top of MTEB Reranking when latency / memory budget allows.\n\n**Minimum dataset:** 500k+ labeled `(query, passage, label)` tuples for production. 10k-100k labeled pairs for continue-training on domain data. Low-resource languages may have less than 10k labeled pairs. In that case, lean on a multilingual base's pretraining and accept a noisier signal.\n\n**\"Small\" multilingual is ~100M+ params**, not 17M-50M like the English small models. mMiniLMv2-L12-H384 (~117M) is roughly the small-end for usable multilingual rerankers.\n\n## [SPARSE] Sparse Encoder (SPLADE)\n\nSPLADE requires a fill-mask / `AutoModelForMaskedLM`-compatible checkpoint. Encoder-only MLM models work out of the box. **Decoder LLMs do not**.\n\n- **Continue from existing SPLADE (English)**: `naver/splade-*` (the canonical family), `opensearch-project/opensearch-neural-sparse-encoding-*` (incl. `-doc-v2-distill`, `-doc-v3-distill` / `-doc-v3-gte`), `prithivida/Splade_PP_en_v*`, `ibm-granite/granite-embedding-30m-sparse`.\n- **Continue from existing SPLADE (multilingual)**: `opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1`.\n- **Fresh-start English** (≥500k pairs): any encoder with an MLM head (`distilbert/distilbert-base-uncased`, `google-bert/bert-base-uncased`). Pure `AutoModel` checkpoints without MLM won't work. Discover MLM bases: `hf models list --filter fill-mask --sort downloads --limit 20`.\n- **Fresh-start multilingual**: `FacebookAI/xlm-roberta-base` (has MLM head). For other multilingual MLM bases: add `--filter `.\n\n**Minimum dataset:** 500k+ triplets (with mined hard negatives) for a competitive SPLADE. 50k+ triplets for domain adaptation on existing SPLADE.\n\n## Cross-cutting tips\n\n- **Non-English retrieval starting points** (when language tag returns 0 results): check `intfloat/multilingual_e5_train_data` for parallel pair data, MIRACL via the `sentence-transformers/miracl` mirror for multilingual retrieval, mMARCO via `unicamp-dl/mmarco` (14 languages, parquet-backed).\n- **Avoid script-based dataset loaders.** `datasets >= 4` rejects them with `RuntimeError: Dataset scripts are no longer supported`. Look for parquet-backed mirrors (e.g. `sentence-transformers/miracl` instead of `miracl/miracl`).\n- **`hf datasets sql` requires DuckDB** (`pip install duckdb`). Without it, fall back to `python -c \"from datasets import load_dataset; ds = load_dataset('', ...); print(ds.column_names, ds[0])\"`.\n","skills/train-sentence-transformers/references/evaluators_multi_vector_encoder.md":"# Multi-Vector-Encoder Evaluators\n\nAll evaluators live in `sentence_transformers.multi_vector_encoder.evaluation`. They mirror the bi-encoder evaluators but use MaxSim scoring end-to-end.\n\n## Top-line pick\n\n| Task | Evaluator |\n|---|---|\n| Retrieval on a suite of common English IR benchmarks | `MultiVectorNanoBEIREvaluator` |\n| Custom retrieval corpus (your own docs / queries / qrels) | `MultiVectorInformationRetrievalEvaluator` |\n| Distillation from a cross-encoder teacher | `MultiVectorDistillationEvaluator` |\n| Reranking a fixed candidate list per query | `MultiVectorRerankingEvaluator` |\n| Triplet accuracy (does anchor score positive > negative?) | `MultiVectorTripletEvaluator` |\n\n**Default recommendation for training**: `MultiVectorNanoBEIREvaluator` on a subset of NanoBEIR datasets (e.g. `[\"msmarco\", \"nq\", \"fiqa2018\"]`) during training, and the full suite at end-of-run. Cheap, well-calibrated, and the metric key format is stable.\n\n## `MultiVectorNanoBEIREvaluator`\n\nRuns the 13 NanoBEIR sub-datasets and returns nDCG@10 averaged across them.\n\n```python\nfrom sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator\n\nevaluator = MultiVectorNanoBEIREvaluator(\n dataset_names=[\"msmarco\", \"nq\", \"fiqa2018\"], # subset for training-time evals\n batch_size=16,\n)\n```\n\n**primary_metric**: `\"NanoBEIR_mean_maxsim_ndcg@10\"` (unless you pass `aggregate_key=\"...\"`, in which case it becomes `\"NanoBEIR_{aggregate_key}_maxsim_ndcg@10\"`). Note that `NanoBEIREvaluator` composes its display name from `aggregate_key` (and optionally `truncate_dim`), it does not accept a `name` kwarg.\n\n**`metric_for_best_model` key**: `\"eval_NanoBEIR_mean_maxsim_ndcg@10\"` (add the `eval_` prefix, since Trainer adds it to evaluator metrics).\n\n- `dataset_names`: list of NanoBEIR sub-datasets. Full list: `msmarco`, `nq`, `fiqa2018`, `hotpotqa`, `nfcorpus`, `arguana`, `scidocs`, `climatefever`, `dbpedia`, `fever`, `quoraretrieval`, `scifact`, `touche2020`. Note it is `quoraretrieval`, not `quora`: an invalid name raises at construction.\n- `batch_size`: also drives corpus encoding, so scale to fit memory.\n- English-only. For non-English retrieval, use `MultiVectorInformationRetrievalEvaluator` with your own corpus.\n\n## `MultiVectorInformationRetrievalEvaluator`\n\nFull IR evaluator over a corpus + queries + qrels you supply.\n\n```python\nfrom sentence_transformers.multi_vector_encoder.evaluation import MultiVectorInformationRetrievalEvaluator\n\nevaluator = MultiVectorInformationRetrievalEvaluator(\n queries={qid: text, ...}, # dict of query_id to query_text\n corpus={did: text, ...}, # dict of doc_id to doc_text\n relevant_docs={qid: {did1, did2, ...}, ...}, # qid to set of relevant doc_ids\n batch_size=16,\n name=\"my-eval\", # optional, prefixes the metric key\n write_csv=True, # persists per-eval metrics\n ndcg_at_k=[10], # customize k-values if you need @20/@100 etc.\n)\n```\n\n**primary_metric**: `\"maxsim_ndcg@10\"` by default (built from `score_function=\"maxsim\"` and `max(ndcg_at_k)`).\n\n**`metric_for_best_model` key** with `name=\"my-eval\"`: `\"eval_my-eval_maxsim_ndcg@10\"`.\n\n- `main_score_function` overrides the score used in `primary_metric` if you set a non-default. Default is `None`, which resolves to `maxsim` from the model's `similarity_fn_name` at call time. For MVE just leave it as `None`.\n- Reports MRR@k, nDCG@k, Recall@k, Precision@k, MAP@k for the k-values you pass (`mrr_at_k`, `ndcg_at_k`, etc.).\n- `chunk_elements` (default `None`) is the element budget for the padded `(chunk, d_tokens, dim)` documents plus the `(num_queries, chunk, q_tokens, d_tokens)` MaxSim scoring intermediate. Leave it as `None`: `maxsim` packs documents into chunks under a 100M-element budget (at most ~400 MB, half that in bf16 / fp16), adapting to the query count and document lengths. Lower it to cut evaluation memory further. It is a separate axis from `corpus_chunk_size`, which caps how many encoded documents are held at a time.\n\n## `MultiVectorDistillationEvaluator`\n\nRegression of student MaxSim scores against teacher scores over a held-out set of `(query, doc, teacher_score)` rows. Use it when you're distilling.\n\n**primary_metric**: `\"spearman\"` (correlation of student scores with teacher scores).\n\n**`metric_for_best_model` key** with `name=\"my-eval\"`: `\"eval_my-eval_spearman\"`.\n\n- Pair with `MultiVectorMarginMSELoss` or `MultiVectorDistillKLDivLoss` during training.\n- Higher is better (student ranking matches teacher).\n\n## `MultiVectorRerankingEvaluator`\n\nReranks a fixed list of candidates per query. Same shape as the bi-encoder `RerankingEvaluator`, MaxSim-scored.\n\n- **Data**: `samples = [{\"query\": ..., \"positive\": [...], \"negative\": [...]} , ...]`.\n- Reports MAP, MRR@10, nDCG@10 by default.\n\n## `MultiVectorTripletEvaluator`\n\nFraction of `(anchor, positive, negative)` triplets where `sim(anchor, positive) > sim(anchor, negative)`. Cheap sanity signal, not what you want to gate a real IR release on.\n\n## Named-evaluator metric key format (universal)\n\nIf you pass `name=\"...\"` to any of these, the metric key format is:\n\n```\neval_{name}_{primary_metric}\n```\n\nIf you don't pass `name`, the key is just `eval_{primary_metric}`. The trainer's `metric_for_best_model=...` must match exactly, or `load_best_model_at_end` selects the wrong checkpoint. **Run the evaluator once before training** to observe the exact key format: `primary_metric` is `None` until the first `evaluator(model)` call, which composes it from the resolved score function and the `ndcg_at_k` you passed, then prefixes `name`. The production template does exactly this before building `TrainingArguments`.\n\n## Gotchas\n\n- **Metric key mismatch on `metric_for_best_model`**: silent failure. Training runs to completion and `load_best_model_at_end` picks a stale checkpoint. Always print `evaluator(model)` output once before starting the trainer and confirm the key format.\n- **Eval-time OOM**: MaxSim scoring builds `(num_queries, chunk, q_tokens, d_tokens)` intermediates, which `maxsim` bounds by default under its 100M-element budget. If you still OOM, lower `chunk_elements`, then drop `batch_size`. NanoBEIR corpora can be small enough that oversized `batch_size` bites before you'd expect.\n- **`MultiVectorNanoBEIREvaluator` with a Non-English base**: it's English-only. You'll get zero or garbage scores. Use `MultiVectorInformationRetrievalEvaluator` with an in-domain corpus.\n- **Distillation eval on the same data as training**: measures memorization, not generalization. Hold out a separate `(query, doc, teacher_score)` split.\n","skills/train-sentence-transformers/SKILL.md":"---\nname: train-sentence-transformers\ndescription: Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder, dense or static embedding model for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker, pair scoring for two-stage retrieval / pair classification), `SparseEncoder` (SPLADE, sparse embedding model for learned-sparse retrieval), and `MultiVectorEncoder` (ColBERT / late-interaction, per-token embeddings scored with MaxSim). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.\n---\n\n# Train a sentence-transformers Model\n\n**This SKILL.md is a router, not a manual.** It tells you which references and example scripts to load for your task. The actual content (recommended losses, evaluators, training-script structure, model selection, training-arg knobs, troubleshooting) lives in `references/` and `scripts/`.\n\n**Do not synthesize a training script from this file alone.** Open the per-type production template (`scripts/train__example.py`) and copy it as your starting point. The templates contain load-bearing scaffolding (autocast helper, model-card class, logger silencing list, `force=True`, `seed`, TF32, version-compatible imports, named-evaluator metric handling) that prior agent runs have repeatedly missed when rolling their own from a synthesized snippet.\n\n## 1. Identify the model type\n\n| Tag | Class | What it does | When to pick |\n|---|---|---|---|\n| **[SentenceTransformer]** | `SentenceTransformer` (bi-encoder) | Maps each input to a fixed-dim dense vector | Retrieval, similarity, clustering, classification, paraphrase mining, dedup |\n| **[CrossEncoder]** | `CrossEncoder` (reranker) | Scores `(query, passage)` pairs jointly | Two-stage retrieval (rerank top-100 from bi-encoder), pair classification |\n| **[SparseEncoder]** | `SparseEncoder` (SPLADE) | Sparse vectors over the vocabulary | Learned-sparse retrieval, inverted-index backends (Elasticsearch / OpenSearch / Lucene) |\n| **[MultiVectorEncoder]** | `MultiVectorEncoder` (ColBERT) | One embedding per token, scored with MaxSim | Late-interaction retrieval, recall gains over bi-encoders at higher storage cost, multimodal (ColPali / ColQwen2) |\n\nTiebreakers when the request is ambiguous: \"embedding model\" / \"vector search\" / \"similarity\" → **[SentenceTransformer]**. \"rerank\" / \"ranker\" / \"two-stage\" → **[CrossEncoder]**. \"SPLADE\" / \"sparse\" / \"inverted index\" → **[SparseEncoder]**. \"ColBERT\" / \"late interaction\" / \"multi-vector\" / \"MaxSim\" / \"ColPali\" / \"ColQwen\" → **[MultiVectorEncoder]**. If still unclear, ask.\n\n## 2. Required reading\n\n**Read these in full before writing any code. Do not triage by perceived relevance.**\n\n### Per-type: always required\n\n**[SentenceTransformer]**\n- `references/losses_sentence_transformer.md`: loss-to-data-shape mapping, `BatchSamplers.NO_DUPLICATES` requirement for MNRL-family, `Cached*` ↔ `gradient_checkpointing` incompatibility.\n- `references/evaluators_sentence_transformer.md`: evaluator-to-task mapping, `metric_for_best_model` key construction (named vs unnamed), per-evaluator `primary_metric` values.\n- `references/model_architectures.md`: encoder vs decoder vs static vs Router pipelines, pooling rules (mean / cls / lasttoken), auto-mean-pooling behavior for fresh-start MLM bases.\n- `scripts/train_sentence_transformer_example.py`: production template. Copy this as your starting point.\n\n**[CrossEncoder]**\n- `references/losses_cross_encoder.md`: pointwise / pairwise / listwise / distillation, `pos_weight` derivation, `activation_fn=Identity()` mandatory for non-BCE losses (silent eval-rank collapse otherwise).\n- `references/evaluators_cross_encoder.md`: `CrossEncoderRerankingEvaluator` recipe, named-evaluator key format `eval_{name}_{primary_metric}`.\n- `scripts/train_cross_encoder_example.py`: production template. Copy this as your starting point.\n\n**[SparseEncoder]**\n- `references/losses_sparse_encoder.md`: `SpladeLoss` wrapper requirement, FLOPS regularizer weights, smoke-test active-dim ramp behavior.\n- `references/evaluators_sparse_encoder.md`: `SparseNanoBEIREvaluator` (English-only) and the in-domain alternative, `eval_{name}_{primary_metric}` key format.\n- `scripts/train_sparse_encoder_example.py`: production template. Copy this as your starting point.\n\n**[MultiVectorEncoder]**\n- `references/losses_multi_vector_encoder.md`: MaxSim scoring, scale choice per scoring mode (`scale=1.0` for MaxSim, roughly the average query length for MeanMaxSim), MNRL / CachedMNRL / MarginMSE / DistillKLDiv, XTR-vs-ColBERT scoring, CachedMNRL ↔ `gradient_checkpointing` incompatibility.\n- `references/evaluators_multi_vector_encoder.md`: `MultiVectorNanoBEIREvaluator` (English-only) and the in-domain alternative, `eval_NanoBEIR_mean_maxsim_ndcg@10` key format, distillation-eval spearman variant.\n- `scripts/train_multi_vector_encoder_example.py`: production template. Copy this as your starting point.\n\n### Cross-cutting: always required (regardless of task)\n\n- `references/training_args.md`: `TrainingArguments` knobs, precision rules (load fp32 + autocast bf16/fp16, never `torch_dtype=bfloat16`), `warmup_steps` (float) vs deprecated `warmup_ratio`, `save_steps` must be a multiple of `eval_steps` for `load_best_model_at_end`, schedulers, HPO, tracker, resume, hub-push variants.\n- `references/dataset_formats.md`: column-matching rules (label name auto-detection, column-order-not-name), reshaping recipes, hard-negative mining options.\n- `references/base_model_selection.md`: discovery commands, per-type model namespaces, ModernBERT-family `max_seq_length=8192` trap, `datasets >= 4` script-loader rejection, non-English starting-point shortcuts.\n- `references/troubleshooting.md`: symptom-indexed failure recipes. Skim the section headings on every run, even a healthy one. The \"Metrics don't improve\" and \"Hub push fails\" entries cover bugs that bite frequently and are cheaper to recognize before they fire than to debug after.\n\n### Cross-cutting: load when applicable\n\n- `references/hardware_guide.md`: VRAM sizing, multi-GPU, FSDP / DeepSpeed, HF Jobs flavors. Required for >24GB models, multi-GPU, or HF Jobs runs.\n- `references/hf_jobs_execution.md`: required when running on HF Jobs.\n- `references/prompts_and_instructions.md`: required when using prompt-tuned bases (E5, BGE, GTE, Qwen3-Embedding, Instructor, Nomic, etc.) or adding `query: ` / `passage: ` style prefixes.\n\n### Variant scripts (open when the task matches)\n- **[SentenceTransformer]** `scripts/train_sentence_transformer__example.py`.\n- **[CrossEncoder]** `scripts/train_cross_encoder__example.py`.\n- **[SparseEncoder]** `scripts/train_sparse_encoder_distillation_example.py`.\n- Hard-negative mining CLI: `scripts/mine_hard_negatives.py`.\n\n## 3. Defaults\n\nOverride only if the user specifies otherwise:\n- **Local execution.** Pitch HF Jobs only if local hardware can't fit the job.\n- **Single run.** After it completes, propose experimentation if the user would benefit (weak/marginal verdict, \"see how high you can push it\" framing, etc.). Iteration rules in `references/training_args.md` (Experimentation section).\n- **Public Hub push at end-of-run, wrapped in try-except.** On HF Jobs (ephemeral env) ALSO enable in-trainer push (`push_to_hub=True` + `hub_strategy=\"every_save\"`). Details in `references/hf_jobs_execution.md`.\n\n## 4. Constraints the produced script must satisfy\n\nThese are non-negotiable contracts. Implementation lives in the production templates and references. Do not reinvent.\n\n- Capture the pre-training evaluator score as `baseline_eval` **before** `trainer.train()`.\n- Emit a single end-of-run line: `VERDICT: WIN|MARGINAL|REGRESSION | score=... | baseline=... | delta=...`. A monitor scrapes for this.\n- Silence `httpx`, `httpcore`, `huggingface_hub`, `urllib3`, `filelock`, `fsspec` to WARNING (otherwise HF download URLs flood the agent's context).\n- Tee logs to `logs/{RUN_NAME}.log`.\n- End with `model.push_to_hub(...)` wrapped in `try/except`.\n- Smoke-test before any long run (`max_steps=1` + tiny dataset slice). The production templates show one common pattern (`SMOKE_TEST` env var).\n- **[CrossEncoder]** Include `EarlyStoppingCallback(patience>=3)`. CE rerankers often peak mid-training and regress.\n- **[SparseEncoder]** Log `query_active_dims` / `corpus_active_dims` on the verdict line. High nDCG with collapsed sparsity is not a win. The keys come back name-prefixed (e.g. `..._query_active_dims`). Use suffix matching to pluck them. See the SPARSE production template for the exact pattern.\n- **[MultiVectorEncoder]** Match `scale` to the scoring mode on any MNRL-family loss: near `1.0` for unnormalized MaxSim (do not copy `scale=20.0` from bi-encoder MNRL), roughly the average query length with length-normalized MeanMaxSim, since each score is divided by its query's token count. `XTRScores` is a train-only `similarity_fct`: the evaluators reject it, so evaluation always scores with MaxSim, including for XTR-trained models.\n\n## 5. Workflow\n\n1. Identify the model type (§1). Ask if ambiguous.\n2. Load the §2 required-reading files for that type.\n3. Open `scripts/train__example.py` and copy it as your starting point.\n4. Replace `MODEL_NAME`, `DATASET_NAME`, `RUN_NAME`, the loss, and the evaluator with the user's task. Cross-check loss/data-shape match against `references/losses_.md`. Cross-check the `metric_for_best_model` key against `references/evaluators_.md` (named evaluators format the key as `eval_{name}_{primary_metric}`).\n5. Smoke-test (`max_steps=1`).\n6. Run.\n7. After the run, append to `logs/experiments.md` and propose iteration if the verdict is weak/marginal.\n\n## Prerequisites\n\n```bash\npip install \"sentence-transformers[train]>=5.0\" # add [train,image] / [audio] / [video] for [SentenceTransformer] multimodal\n # [MultiVectorEncoder] requires >=6.0\npip install trackio # optional tracker (or wandb / tensorboard / mlflow)\nhf auth login # or set HF_TOKEN with write scope (for Hub push)\n```\n\nGPU strongly recommended. CPU works only for demos and `[SentenceTransformer]` `StaticEmbedding`.\n","skills/train-sentence-transformers/references/evaluators_sparse_encoder.md":"# Evaluators (Sparse Encoder)\n\nAll sparse-encoder evaluators live in `sentence_transformers.sparse_encoder.evaluation`. They mirror the bi-encoder versions with a `Sparse` prefix and default to **dot product** similarity (cosine on sparse vectors is less meaningful).\n\n## Choosing the right evaluator\n\n| Task | Evaluator |\n|---|---|\n| Retrieval (nDCG, MRR, Recall), fast default | `SparseNanoBEIREvaluator` |\n| Retrieval on your own corpus / qrels | `SparseInformationRetrievalEvaluator` |\n| STS / continuous similarity | `SparseEmbeddingSimilarityEvaluator` |\n| Binary classification | `SparseBinaryClassificationEvaluator` |\n| Triplet accuracy | `SparseTripletEvaluator` |\n| Reranking (from retrieval candidates) | `SparseRerankingEvaluator` |\n| MSE vs. teacher (distillation) | `SparseMSEEvaluator` |\n| Translation (cross-lingual alignment) | `SparseTranslationEvaluator` |\n| Hybrid BM25 + sparse retrieval | `ReciprocalRankFusionEvaluator` |\n\nWrap multiple in `SequentialEvaluator` (from `sentence_transformers.base.evaluation`):\n\n```python\nfrom sentence_transformers.base.evaluation import SequentialEvaluator\nevaluator = SequentialEvaluator([sparse_nano_beir, my_custom_ir])\n```\n\n## The default: `SparseNanoBEIREvaluator`\n\nSmall, fast subset of BEIR adapted for sparse retrieval. Typical runtime <1 minute on a mid-range GPU.\n\n```python\nfrom sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator\n\nevaluator = SparseNanoBEIREvaluator(\n dataset_names=[\"msmarco\", \"nfcorpus\", \"nq\"], # default: all 13 NanoBEIR datasets\n batch_size=32,\n show_progress_bar=False,\n)\n```\n\nOutput key for `metric_for_best_model`: **`eval_NanoBEIR_mean_dot_ndcg@10`** (sparse defaults to dot product).\n\n### Sparsity tracking\n\nUnlike the dense variant, the sparse evaluator also reports **active dimension counts** so you can monitor sparsity during training:\n\n- `query_active_dims`: non-zero entries per query vector\n- `document_active_dims`: non-zero entries per document vector\n\nA healthy SPLADE checkpoint typically shows ~30-50 active dims for queries and ~150-250 for documents. If these drift toward the vocab size (~30k), the FLOPS regularization isn't doing its job. Raise `query_regularizer_weight` / `document_regularizer_weight` in `SpladeLoss`.\n\n## Retrieval on your own corpus\n\n### `SparseInformationRetrievalEvaluator`\n\nSame shape as the dense version but operates on sparse vectors internally:\n\n```python\nfrom sentence_transformers.sparse_encoder.evaluation import SparseInformationRetrievalEvaluator\n\nevaluator = SparseInformationRetrievalEvaluator(\n queries={qid: text for qid, text in ...},\n corpus={doc_id: text for doc_id, text in ...},\n relevant_docs={qid: {doc_id, ...} for qid in ...},\n name=\"my-sparse-ir\",\n ndcg_at_k=[10],\n mrr_at_k=[10],\n accuracy_at_k=[1, 5, 10],\n map_at_k=[100],\n batch_size=32,\n)\n```\n\nOutput keys: `eval_{name}_dot_ndcg@10`, `eval_{name}_dot_mrr@10`, etc. Also reports active-dims.\n\nHeavy for large corpora. Use `SparseNanoBEIREvaluator` during training. Reserve full IR for post-training.\n\n## Hybrid retrieval\n\n### `ReciprocalRankFusionEvaluator`\n\nMeasures the performance of combining your sparse encoder with BM25 (or any other retriever) via reciprocal-rank fusion. Useful when shipping a hybrid system is the actual deployment target.\n\n## Other sparse evaluators\n\n### `SparseEmbeddingSimilarityEvaluator`\n\nSTS-style. Computes Pearson/Spearman between sparse vector similarities and gold labels. Uses dot product by default.\n\n### `SparseBinaryClassificationEvaluator`\n\nFor labeled pair classification with sparse embeddings.\n\n### `SparseTripletEvaluator`\n\nFor `(anchor, positive, negative)` triplets. Reports fraction where the positive is closer than the negative (by dot product).\n\n### `SparseRerankingEvaluator`\n\nFor custom re-ranking with sparse embeddings. Same semantics as the dense `RerankingEvaluator`.\n\n### `SparseMSEEvaluator`\n\nFor distillation setups. Compares sparse student embeddings against teacher outputs.\n\n### `SparseTranslationEvaluator`\n\nFor cross-lingual / `make_multilingual`-style alignment checking with sparse embeddings.\n\n## Writing `metric_for_best_model`\n\nPattern: `f\"eval_{evaluator.primary_metric}\"`. Inspect after construction: `print(evaluator.primary_metric)`. Common values:\n- `eval_NanoBEIR_mean_dot_ndcg@10`: `SparseNanoBEIREvaluator` default\n- `eval_{name}_dot_ndcg@10`: `SparseInformationRetrievalEvaluator`\n- `eval_{name}_spearman_dot`: `SparseEmbeddingSimilarityEvaluator`\n\n## Gotchas\n\n- **Always run `evaluator(model)` once before training**. This confirms the pipeline works (a fill-mask base scores ~0 on retrieval until trained).\n- Sparse evaluators default to dot product. Cosine on sparse vectors isn't meaningful.\n- Don't compare dense and sparse metrics directly: different scales (cosine ∈ [-1, 1] vs. dot ∈ [0, ∞)).\n- Always check `query_active_dims` / `document_active_dims`: thousands of active dims per doc means the FLOPS regularizer is mistuned, even if nDCG looks OK.\n","skills/train-sentence-transformers/references/hardware_guide.md":"# Hardware Guide\n\nTraining embedding models is memory-bound more often than compute-bound.\n\n## If you hit OOM\n\nTry in this order:\n\n1. **Reduce `per_device_train_batch_size`**. Raise `gradient_accumulation_steps` to keep the effective batch size for regression losses. (For MNRL, effective batch via grad-accum is **not** equivalent. See point 3.)\n2. **Enable `gradient_checkpointing=True`**. ~30% slower, ~40% less activation memory. Incompatible with `Cached*` losses.\n3. **Switch to a `Cached*` loss**:\n - `CachedMultipleNegativesRankingLoss(model, mini_batch_size=32)`: forwards in mini-batches, accumulates the contrastive loss over the full batch. Can simulate batch sizes of 1024+ on a 24GB GPU.\n - `CachedSpladeLoss(model, loss=..., mini_batch_size=16)`: same trick for sparse.\n - `CachedGISTEmbedLoss(model, guide_model, mini_batch_size=32)`: GIST variant.\n4. **Enable PEFT / LoRA** for decoder models >1B. `LoraConfig(r=64, lora_alpha=128, task_type=\"FEATURE_EXTRACTION\")`. See `../scripts/train_sentence_transformer_with_lora_example.py` (docstring covers when to use, hyperparams, QLoRA, sharing).\n5. **Move to multi-GPU**. See below.\n6. **Shorten sequences**. If truncating to 128 is already sufficient for your task, set `max_seq_length` on the transformer module.\n\n## Multi-GPU\n\n`sentence-transformers` uses `accelerate` under the hood. Distributed training works without code changes.\n\n### Data parallel (DDP)\n\nLaunch:\n\n```bash\naccelerate launch train.py\n\n# or explicitly:\naccelerate launch --multi_gpu --num_processes=4 train.py\n```\n\n`per_device_train_batch_size` stays per-GPU. Effective batch size scales linearly. MNRL's in-batch negatives remain **per-device**, not global, unless you pass `gather_across_devices=True` to losses that support it (`MultipleNegativesRankingLoss`, `CachedMultipleNegativesRankingLoss`, the symmetric variants, `GISTEmbedLoss`, `CachedGISTEmbedLoss`, `SparseMultipleNegativesRankingLoss`).\n\n### FSDP / DeepSpeed\n\nFor models >3B, use `accelerate config` to enable FSDP or DeepSpeed ZeRO. Both are supported: `sentence-transformers` doesn't require any code changes, only the launch config.\n\n```bash\naccelerate config # interactive; choose FSDP or DeepSpeed\naccelerate launch train.py\n```\n\nWith FSDP full-shard: a 7B model trains on 4×24GB GPUs that would OOM on any single one of them.\n\n**FSDP caveats** (from the [distributed training docs](https://sbert.net/docs/sentence_transformer/training/distributed.html)):\n- **Evaluators don't run under FSDP** as of writing: the eval hooks call `model.encode()` which FSDP-wrapped modules can't service mid-training. Plan to evaluate post-training on a single-GPU load of the final checkpoint instead, or train with DDP if you need mid-training evaluation.\n- **Layer wrapping must be specified**, e.g. `fsdp_config={\"transformer_layer_cls_to_wrap\": \"BertLayer\"}` (substitute the right layer class for your model: `BertLayer`, `LlamaDecoderLayer`, `Qwen2DecoderLayer`, etc.). Without this FSDP sharding can silently misbehave.\n- **Slower than DDP** for models that fit on a single GPU. Only reach for FSDP when you actually need the memory savings.\n\nDeepSpeed ZeRO-2/3 is an alternative with its own config. It works identically at the `accelerate config` level.\n\n## Effective batch size for contrastive losses\n\nFor `MultipleNegativesRankingLoss` and its variants, **batch size is a quality knob**, not just a speed knob. Bigger batches = more in-batch negatives = richer gradients.\n\nThe effective pool of in-batch negatives per anchor:\n\n| Setup | In-batch negatives per anchor |\n|---|---|\n| Single GPU, batch 64 | 63 |\n| 4× DDP, per-device batch 64 | 63 local only by default. 255 with `MultipleNegativesRankingLoss(model, gather_across_devices=True)` |\n| Single GPU, CachedMNRL, mini_batch 32, batch 256 | 255 |\n| 4× DDP, CachedMNRL, per-device 256 | 255 local. 1023 with `gather_across_devices=True` |\n\nFor large corpora (retrieval), push toward 512+ effective negatives. For small, clean datasets (STS), 64 is plenty.\n\n## Precision choice by GPU\n\n| GPU generation | Recommended |\n|---|---|\n| T4, V100, GTX 1xxx, RTX 2xxx | `fp16=True` |\n| RTX 3xxx, A10G, A100, L4 | `bf16=True` |\n| RTX 4xxx, H100, B200 | `bf16=True` (or fp8 on H100 via specific kernels, not default) |\n| Apple M-series / ROCm | MPS/ROCm support is variable. `fp16` or `fp32` most reliable |\n\nbf16 is more numerically stable and almost always preferred when available.\n\n## Hugging Face Jobs flavor guide\n\nHugging Face Jobs requires a Pro/Team/Enterprise plan. Pricing is approximate and subject to change. See the [Jobs pricing page](https://huggingface.co/docs/huggingface_hub/guides/jobs).\n\n| Flavor | Memory | Typical use | Est. $/hr |\n|---|---|---|---|\n| `cpu-basic` | ~2 GB | Dataset prep, validation, hard-neg mining (small) | <$0.10 |\n| `cpu-upgrade` | ~4 GB | Same, slightly bigger | $0.10 |\n| `t4-small` | 16 GB | Demos, MiniLM/DistilBERT with small batches | ~$0.75 |\n| `t4-medium` | 16 GB | MiniLM / DistilBERT with larger batch | ~$1.50 |\n| `l4x1` | 24 GB | BERT-base, MPNet, ModernBERT-base | ~$2.50 |\n| `a10g-small` | 24 GB | BERT-base to BERT-large | ~$3.50 |\n| `a10g-large` | 48 GB | ModernBERT-large, Qwen3-0.6B | ~$5.00 |\n| `a10g-largex2` | 96 GB (2× 48GB) | Mid-size multi-GPU | ~$10 |\n| `a100-large` | 80 GB | Large models or big contrastive batches | ~$10-12 |\n| `h100` | 80 GB | Biggest single-GPU | ~$12 |\n| `h100x8` | 640 GB | LLM-scale distributed | ~$96 |\n\nDefaults by base model:\n- MiniLM / DistilBERT -> `t4-small`\n- BERT-base / MPNet / ModernBERT-base -> `a10g-small` or `l4x1`\n- BERT-large / ModernBERT-large -> `a10g-large`\n- Qwen3-0.6B decoder base -> `a10g-large`\n- 1B+ decoder bases with LoRA -> `a10g-large` or `a100-large`\n\nAlways start one flavor **smaller** than you think you need: OOM on Jobs is cheap ($0.50-$5 for a failed run). Underprovisioned is better than overprovisioned for the first attempt. When budgeting `timeout`, add **20-30% buffer** for model loading, checkpoint saving, and Hub push.\n","skills/train-sentence-transformers/references/hf_jobs_execution.md":"# Hugging Face Jobs Execution\n\nRun training on Hugging Face's managed GPUs without provisioning any local infrastructure. The same training script runs locally and on Jobs. This reference covers only the Jobs-specific concerns.\n\n## Prerequisites\n\n- Hugging Face account with a **Pro, Team, or Enterprise** plan. Jobs are paid.\n- `HF_TOKEN` with **write** permission. Log in once locally with `hf auth login` (from the `hf` CLI).\n- Access to the `hf_jobs()` MCP tool, or the `hf` CLI (`curl -LsSf https://hf.co/cli/install.sh | bash -s`).\n\n## The three submission paths\n\n### 1. Inline script via MCP (recommended in Claude Code)\n\nPass the full training script as `script`. Dependencies come from the PEP 723 header.\n\n```python\nhf_jobs(\"uv\", {\n \"script\": \"\"\"\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"sentence-transformers[train]>=5.0\", \"trackio\"]\n# ///\n\n# \n\"\"\",\n \"flavor\": \"a10g-large\",\n \"timeout\": \"3h\",\n \"secrets\": {\"HF_TOKEN\": \"$HF_TOKEN\"},\n})\n```\n\n### 2. Script-from-URL via MCP\n\nUpload the script to the Hub (as a model or dataset repo file) or a Gist, then reference by URL:\n\n```python\nhf_jobs(\"uv\", {\n \"script\": \"https://huggingface.co/USERNAME/scripts/resolve/main/train_bi_encoder.py\",\n \"flavor\": \"a10g-large\",\n \"timeout\": \"3h\",\n \"secrets\": {\"HF_TOKEN\": \"$HF_TOKEN\"},\n})\n```\n\nLocal file paths (`./train.py`, `/path/to/train.py`) **do not work**. Jobs run in isolated containers without access to your filesystem.\n\n### 3. CLI\n\n```bash\nhf jobs uv run \\\n --flavor a10g-large \\\n --timeout 3h \\\n --secrets HF_TOKEN \\\n \"https://huggingface.co/USERNAME/scripts/resolve/main/train.py\"\n```\n\nSyntax gotchas:\n- Command order is `hf jobs uv run`, **not** `hf jobs run uv`.\n- Flags (`--flavor`, `--timeout`, `--secrets`) go **before** the script URL.\n- `--secrets` (plural), not `--secret`.\n\n## Required script modifications for Jobs\n\nAdd these to your `TrainingArguments`:\n\n```python\nargs = SentenceTransformerTrainingArguments(\n ...,\n push_to_hub=True,\n hub_model_id=\"your-username/my-model\",\n hub_strategy=\"every_save\", # push each checkpoint; timeout-safe\n save_strategy=\"steps\",\n save_steps=0.1, # 10 saves/pushes per epoch; scales with dataset size\n)\n```\n\nWhy each matters:\n\n| Argument | Why |\n|---|---|\n| `push_to_hub=True` | The Jobs container is destroyed after the job finishes. Without Hub push, all weights are lost. |\n| `hub_model_id` | Required to identify the destination repo. |\n| `hub_strategy=\"every_save\"` | Default, but worth being deliberate about on Jobs: each checkpoint is pushed as it's written, so a timeout leaves all completed checkpoints on the Hub. `\"end\"` only pushes once `trainer.train()` returns, so a timeout loses everything. |\n| `save_strategy=\"steps\"` + `save_steps=0.1` | Checkpoints must actually be saved for `hub_strategy=\"every_save\"` to push them. Fractional `0.1` = save every 10% of training, auto-scales with dataset size. |\n\n## Secrets\n\nSecrets are environment variables injected into the Jobs container. They never appear in logs and are not part of the script.\n\n| Secret | Required when |\n|---|---|\n| `HF_TOKEN` | Always, for Hub push. Also covers Trackio auth. |\n| `WANDB_API_KEY` | Using `report_to=\"wandb\"`. |\n| `MLFLOW_TRACKING_URI`, `MLFLOW_TRACKING_TOKEN` | Using MLflow with a remote server. |\n\nThe `$HF_TOKEN` syntax in the job config references the value from your local environment at submission time. The literal string `$HF_TOKEN` is replaced with your token's value. Never hardcode tokens in the script itself.\n\nTrackio (the default tracker in this skill) uses `HF_TOKEN` for auth, so no extra secrets are needed. Only switch to the W&B / MLflow rows above if you're using those trackers.\n\n## Timeout\n\nDefault is **30 minutes**, which is too short for almost any real training. Set explicitly:\n\n```python\n\"timeout\": \"2h\" # 2 hours\n\"timeout\": \"90m\" # 90 minutes\n\"timeout\": \"1.5h\" # 90 minutes\n\"timeout\": 7200 # seconds, as integer\n```\n\nRule: **estimated training time × 1.3**. The extra buffer covers model loading, dataset caching, checkpoint saving, and Hub push.\n\nOn timeout, the container is killed immediately. Only data on the Hub (`hub_strategy=\"every_save\"` saves you here) or in persistent volumes survives.\n\n## Dataset caching\n\nHugging Face datasets are cached at `~/.cache/huggingface/datasets` by default. That's **inside the container**, which is destroyed after the job. Each Jobs run re-downloads the dataset.\n\nFor large datasets (>5 GB), this matters. Options:\n\n- **Persistent `/data` volume** (Jobs feature, check current documentation): set `HF_DATASETS_CACHE=/data/datasets` so caches persist across jobs.\n- **Pre-cache locally, push to Hub**: if the dataset is on Hub already, nothing to do. If it's local-only, `dataset.push_to_hub(...)` once so subsequent jobs load from Hub.\n\n## Monitoring a running job\n\n```bash\nhf jobs ps [--all] # running (or all) jobs\nhf jobs inspect # full config + status\nhf jobs logs [--follow|--tail N] # tail or stream\nhf jobs cancel \nhf jobs hardware # list flavors + hourly rates\n```\n\n`hf jobs logs --follow` under `Bash run_in_background` pairs nicely with a `Monitor` watching for the `VERDICT:` line emitted by your training script's verdict block.\n\nMCP equivalents (signatures may vary by server version, so check the actual\ntool listing): `hf_jobs(\"ps\")`, `hf_jobs(\"logs\", {\"job_id\": ...})`,\n`hf_jobs(\"cancel\", {\"job_id\": ...})`.\n\nFor recurring runs, `hf jobs scheduled uv run \"\"