Technical Documentation: FlagOpen/FlagEmbedding

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

⚡️BGE: One-Stop Retrieval Toolkit For Search and RAG

text Copy

<a href=#news>News</a> |
    <a href=#installation>Installation</a> |
    <a href=#quick-start>Quick Start</a> |
    <a href=#community>Community</a> |
    [Projects](https://github.com/FlagOpen/FlagEmbedding/tree/master/research) |
    <a href=#model-list>Model List</a> |
    [Contributor](#contributors) |
    [Citation](#citation) |
    [License](#license)

English | 中文

News Copy Chapter

• 3/6/2025: :fire::fire: Introduce BGE-VL (HF repo), State-Of-The-Art multimodal embedding models to support Any visual search applications (everything, including text-to-image, image-to-text, image&prompt-to-image, text-to-image&text, and more)! They are released under the MIT license and are completely free for both academic and commercial use. We also release MegaPairs (repo, paper), a massive synthetic dataset which empowers BGE-VL!

• 12/5/2024: :book: We built the BGE documentation for centralized BGE information and materials!

• 10/29/2024: :earth_asia: We created WeChat group for BGE. Scan the QR code to join the group chat! To get the first hand message about our updates and new release, or having any questions or ideas, join us now!

• 10/22/2024: We release another interesting model: OmniGen, which is a unified image generation model supporting various tasks. OmniGen can accomplish complex image generation tasks without the need for additional plugins like ControlNet, IP-Adapter, or auxiliary models such as pose detection and face detection.

• 9/10/2024: Introducing MemoRAG, a step forward towards RAG 2.0 on top of memory-inspired knowledge discovery (repo: https://github.com/qhjqhj00/MemoRAG, paper: https://arxiv.org/pdf/2409.05591v1)

• 9/2/2024: Start to maintain the tutorials. The contents within will be actively updated and eariched, stay tuned! :books:

• 7/26/2024: Release a new embedding model bge-en-icl, an embedding model that incorporates in-context learning capabilities, which, by providing task-relevant query-response examples, can encode semantically richer queries, further enhancing the semantic representation ability of the embeddings.

• 7/26/2024: Release a new embedding model bge-multilingual-gemma2, a multilingual embedding model based on gemma-2-9b, which supports multiple languages and diverse downstream tasks, achieving new SOTA on multilingual benchmarks (MIRACL, MTEB-fr, and MTEB-pl).

• 7/26/2024: Release a new lightweight reranker bge-reranker-v2.5-gemma2-lightweight, a lightweight reranker based on gemma-2-9b, which supports token compression and layerwise lightweight operations, can still ensure good performance while saving a significant amount of resources. :fire:

  More

• 6/7/2024: Release a new benchmark MLVU, the first comprehensive benchmark specifically designed for long video understanding. MLVU features an extensive range of video durations, a diverse collection of video sources, and a set of evaluation tasks uniquely tailored for long-form video understanding. :fire:

• 5/21/2024: Release a new benchmark AIR-Bench together with Jina AI, Zilliz, HuggingFace, and other partners. AIR-Bench focuses on a fair out-of-distribution evaluation for Neural IR & RAG. It generates the synthetic data for benchmarking w.r.t. diverse domains and languages. It is dynamic and will be updated on regular basis. Leaderboard :fire:

• 4/30/2024: Release Llama-3-8B-Instruct-80K-QLoRA, extending the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA training on a few synthesized long-context data. The model achieves remarkable performance on various long-context benchmarks. Code :fire:

• 3/18/2024: Release new rerankers, built upon powerful M3 and LLM (GEMMA and MiniCPM, not so large actually :smiley:) backbones, supporitng multi-lingual processing and larger inputs, massive improvements of ranking performances on BEIR, C-MTEB/Retrieval, MIRACL, LlamaIndex Evaluation :fire:

• 3/18/2024: Release Visualized-BGE, equipping BGE with visual capabilities. Visualized-BGE can be utilized to generate embeddings for hybrid image-text data. :fire:

• 1/30/2024: Release BGE-M3, a new member to BGE model series! M3 stands for Multi-linguality (100+ languages), Multi-granularities (input length up to 8192), Multi-Functionality (unification of dense, lexical, multi-vec/colbert retrieval).
  It is the first embedding model which supports all three retrieval methods, achieving new SOTA on multi-lingual (MIRACL) and cross-lingual (MKQA) benchmarks.
  Technical Report and Code. :fire:

• 1/9/2024: Release Activation-Beacon, an effective, efficient, compatible, and low-cost (training) method to extend the context length of LLM. Technical Report

• 12/24/2023: Release LLaRA, a LLaMA-7B based dense retriever, leading to state-of-the-art performances on MS MARCO and BEIR. Model and code will be open-sourced. Please stay tuned. Technical Report and Code

• 11/23/2023: Release LM-Cocktail, a method to maintain general capabilities during fine-tuning by merging multiple language models. Technical Report

• 10/12/2023: Release LLM-Embedder, a unified embedding model to support diverse retrieval augmentation needs for LLMs. Technical Report

• 09/15/2023: The technical report of BGE has been released

• 09/15/2023: The massive training data of BGE has been released

• 09/12/2023: New models:

  • New reranker model: release cross-encoder models BAAI/bge-reranker-base and BAAI/bge-reranker-large, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
  • update embedding model: release bge-*-v1.5 embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.

• 09/07/2023: Update fine-tune code: Add script to mine hard negatives and support adding instruction during fine-tuning.

• 08/09/2023: BGE Models are integrated into Langchain, you can use it like this; C-MTEB leaderboard is available.

• 08/05/2023: Release base-scale and small-scale models, best performance among the models of the same size 🤗

• 08/02/2023: Release bge-large-*(short for BAAI General Embedding) Models, rank 1st on MTEB and C-MTEB benchmark! :tada: :tada:

• 08/01/2023: We release the Chinese Massive Text Embedding Benchmark (C-MTEB), consisting of 31 test dataset.

BGE (BAAI General Embedding) focuses on retrieval-augmented LLMs, consisting of the following projects currently:

• Inference: Embedder, Reranker
• Finetune: Embedder, Reranker
• Evaluation
• Dataset
• Tutorials
• research

Installation Copy Chapter

Using pip: Copy Section

If you do not want to finetune the models, you can install the package without the finetune dependency:

pip install -U FlagEmbedding

If you want to finetune the models, you can install the package with the finetune dependency:

pip install -U FlagEmbedding[finetune]

Install from sources: Copy Section

Clone the repository and install

git clone https://github.com/FlagOpen/FlagEmbedding.git
cd FlagEmbedding
# If you do not need to finetune the models, you can install the package without the finetune dependency:
pip install  .
# If you want to finetune the models, install the package with the finetune dependency:
# pip install  .[finetune]

For development in editable mode:

# If you do not need to finetune the models, you can install the package without the finetune dependency:
pip install -e .
# If you want to finetune the models, install the package with the finetune dependency:
# pip install -e .[finetune]

Quick Start Copy Chapter

First, load one of the BGE embedding model:

from FlagEmbedding import FlagAutoModel

model = FlagAutoModel.from_finetuned('BAAI/bge-base-en-v1.5',
                                      query_instruction_for_retrieval="Represent this sentence for searching relevant passages:",
                                      use_fp16=True)

Then, feed some sentences to the model and get their embeddings:

sentences_1 = ["I love NLP", "I love machine learning"]
sentences_2 = ["I love BGE", "I love text retrieval"]
embeddings_1 = model.encode(sentences_1)
embeddings_2 = model.encode(sentences_2)

Once we get the embeddings, we can compute similarity by inner product:

similarity = embeddings_1 @ embeddings_2.T
print(similarity)

For more details, you can refer to embedder inference, reranker inference, embedder finetune, reranker fintune, evaluation.

If you're unfamiliar with any of related concepts, please check out the tutorial. If it's not there, let us know.

For more interesting topics related to BGE, take a look at research.

Community Copy Chapter

We are actively maintaining the community of BGE and FlagEmbedding. Let us know if you have any suggessions or ideas!

Currently we are updating the tutorials, we aim to create a comprehensive and detailed tutorial for beginners on text retrieval and RAG. Stay tuned!

The following contents are releasing in the upcoming weeks:

• Evaluation

• BGE-EN-ICL

  The whole tutorial roadmap

Model List Copy Chapter

bge is short for BAAI general embedding.

Contributors: Copy Section

Thank all our contributors for their efforts and warmly welcome new members to join in!

Citation Copy Chapter

If you find this repository useful, please consider giving a star :star: and citation

@misc{bge_m3,
  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng},
  year={2023},
  eprint={2309.07597},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

@misc{cocktail,
      title={LM-Cocktail: Resilient Tuning of Language Models via Model Merging}, 
      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Xingrun Xing},
      year={2023},
      eprint={2311.13534},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{llm_embedder,
      title={Retrieve Anything To Augment Large Language Models}, 
      author={Peitian Zhang and Shitao Xiao and Zheng Liu and Zhicheng Dou and Jian-Yun Nie},
      year={2023},
      eprint={2310.07554},
      archivePrefix={arXiv},
      primaryClass={cs.IR}
}

@misc{bge_embedding,
      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff},
      year={2023},
      eprint={2309.07597},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License Copy Chapter

FlagEmbedding is licensed under the MIT License.

README (`docs/README.md`) Copy Section

install the required pkgs:

pip install -r requirements.txt

to host the webpages locally:

python -m http.server

Source C MTEB (`docs/source/C-MTEB.rst`) Copy Section

.. C-MTEB
.. ======

.. Introduction
.. ------------

.. C-MTEB <https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB>_ is a benchmark for chinese text embedding. It contains 35
.. datasets in 6 different tasks, providing a comprehensive evaluation to the quality of an embedding model on Chinese.

.. .. image:: ../_static/img/C_MTEB.png
.. :width: 700
.. :align: center

.. Installation
.. ------------

.. C-MTEB is developed based on MTEB, you can install C-MTEB by:

.. .. code:: bash

.. pip install -U C_MTEB

.. or install by FlagEmbedding's repo:

.. .. code:: bash

.. git clone https://github.com/FlagOpen/FlagEmbedding.git
.. cd FlagEmbedding/C_MTEB
.. pip install -e .

.. Citing the Work
.. ---------------

.. There are more details in our publication. If you find C-MTEB useful, you can cite it by:

.. .. code::

.. @misc{c-pack,
.. title={C-Pack: Packaged Resources To Advance General Chinese Embedding},
.. author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff},
.. year={2023},
.. eprint={2309.07597},
.. archivePrefix={arXiv},
.. primaryClass={cs.CL}
.. }

Source Index (`docs/source/index.rst`) Copy Section

.. FlagEmbedding documentation master file, created by
sphinx-quickstart on Sat Oct 12 13:27:49 2024.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.

:html_theme.sidebar_secondary.remove: True

Welcome to BGE!

.. Welcome to BGE documentation!

.. figure:: _static/img/bge_panda.jpg
:width: 400
:align: center

.. grid:: 3
:gutter: 3

.. grid-item-card:: :octicon:milestone Introduction

New to BGE? Quickly get hands-on information.

  +++

  .. button-ref:: Introduction/index
     :expand:
     :color: primary
     :click-parent:

     To Introduction

.. grid-item-card:: :octicon:package BGE Models

Get to know BGE embedding models and rerankers.

  +++

  .. button-ref:: bge/index
     :expand:
     :color: primary
     :click-parent:

     To BGE

.. grid-item-card:: :octicon:log Tutorials

Find useful tutorials to start with if you are looking for guidance

  +++

  .. button-ref:: tutorial/index
     :expand:
     :color: primary
     :click-parent:

     To Tutorials

.. grid-item-card:: :octicon:codescan API

Check the API of classes and functions in FlagEmbedding.

  +++

  .. button-ref:: API/index
     :expand:
     :color: primary
     :click-parent:

     To APIs

.. grid-item-card:: :octicon:question FAQ

Take a look of questions people frequently asked.

  +++

  .. button-ref:: FAQ/index
     :expand:
     :color: primary
     :click-parent:

     To FAQ

.. grid-item-card:: :octicon:people Community

Welcome to join BGE community!

  +++

  .. button-ref:: community/index
     :expand:
     :color: primary
     :click-parent:

     To Community

Besides the resources we provide here in this documentation, please visit our GitHub repo <https://github.com/FlagOpen/FlagEmbedding>_ for more contents including:

• Want to get familiar with BGE quickly? There are hands-on examples <https://github.com/FlagOpen/FlagEmbedding/tree/068e86f58eccc3107aacb119920de8dba9caa913/examples>_ to run for embedder and reranker's inference, evaluation, and finetuning.
• Unfamiliar with some area, keywords or techniques of retrieval and RAG? We provide tutorials <https://github.com/FlagOpen/FlagEmbedding/tree/068e86f58eccc3107aacb119920de8dba9caa913/Tutorials>_ to teach you basic knowledge and coding tips.
• Interested in research topics that expanding from BGE and retrieval? Our research <https://github.com/FlagOpen/FlagEmbedding/tree/068e86f58eccc3107aacb119920de8dba9caa913/research>_ folder contains many exciting topics for you to explore.

BGE is developed by Beijing Academy of Artificial Intelligence (BAAI).

|

.. image:: _static/img/BAAI_logo.png
:target: https://github.com/FlagOpen/FlagEmbedding
:width: 300
:align: center

.. toctree::
:maxdepth: 1
:hidden:

Home

.. toctree::
:hidden:
:maxdepth: 1
:caption: Introduction

Introduction/index

.. toctree::
:hidden:
:maxdepth: 1
:caption: BGE

bge/index

.. toctree::
:hidden:
:maxdepth: 2
:caption: Tutorials

tutorial/index

.. toctree::
:hidden:
:maxdepth: 5
:caption: API

API/index

.. toctree::
:hidden:
:maxdepth: 1
:caption: FAQ

FAQ/index

.. toctree::
:hidden:
:maxdepth: 1
:caption: Community

community/index

Source API Abc (`docs/source/API/abc.rst`) Copy Section

Abstract Class

.. toctree::
abc/inference
abc/evaluation
abc/finetune

Source API Evaluation (`docs/source/API/evaluation.rst`) Copy Section

Evaluation

.. toctree::
evaluation/mteb
evaluation/airbench
evaluation/msmarco
evaluation/beir
evaluation/miracl
evaluation/mkqa
evaluation/mldr

Source API Index (`docs/source/API/index.rst`) Copy Section

API

.. toctree::
:maxdepth: 1

abc
inference
evaluation
finetune

Source API Inference (`docs/source/API/inference.rst`) Copy Section

Inference

.. toctree::
inference/FlagAutoModel
inference/FlagAutoReranker
inference/embedder/embedder
inference/reranker/reranker

Source FAQ Index (`docs/source/FAQ/index.rst`) Copy Section

FAQ

Below are some commonly asked questions.

.. tip::

For more questions, search in issues on GitHub or join our community!

.. dropdown:: Having network issue when connecting to Hugging Face?
:animate: fade-in-slide-down

Try to set the :code:`HF_ENDPOINT` to `HF mirror <https://hf-mirror.com/>`_ instead.

.. code:: bash

    export HF_ENDPOINT=https://hf-mirror.com

.. dropdown:: When does the query instruction need to be used?
:animate: fade-in-slide-down

For a retrieval task that uses short queries to find long related documents, it is recommended to add instructions for these short queries. 
The best method to decide whether to add instructions for queries is choosing the setting that achieves better performance on your task. 
In all cases, the documents/passages do not need to add the instruction.

.. dropdown:: Why it takes quite long to just encode 1 sentence?
:animate: fade-in-slide-down

Note that if you have multiple CUDA GPUs, FlagEmbedding will automatically use all of them. 
Then the time used to start the multi-process will cost way longer than the actual encoding.
Try to just use CPU or just single GPU for simple tasks.

.. dropdown:: The embedding results are different for CPU and GPU?
:animate: fade-in-slide-down

The encode function will use FP16 by default if GPU is available, which leads to different precision. 
Set :code:`fp16=False` to get full precision.

.. dropdown:: How many languages do the multi-lingual models support?
:animate: fade-in-slide-down

The training datasets cover up to 170+ languages. 
But note that due to the unbalanced distribution of languages, the performances will be different.
Please further test refer to the real application scenario.

.. dropdown:: How does the different retrieval method works in bge-m3?
:animate: fade-in-slide-down

- Dense retrieval: map the text into a single embedding, e.g., `DPR <https://arxiv.org/abs/2004.04906>`_, `BGE-v1.5 <../bge/bge_v1_v1.5>`_
- Sparse retrieval (lexical matching): a vector of size equal to the vocabulary, with the majority of positions set to zero, calculating a weight only for tokens present in the text. e.g., BM25, `unicoil <https://arxiv.org/pdf/2106.14807>`_, and `splade <https://arxiv.org/abs/2107.05720>`_
- Multi-vector retrieval: use multiple vectors to represent a text, e.g., `ColBERT <https://arxiv.org/abs/2004.12832>`_.

.. dropdown:: Recommended vector database?
:animate: fade-in-slide-down

Generally you can use any vector database (open-sourced, commercial). We use `Faiss <https://github.com/facebookresearch/faiss>`_ by default in our evaluation pipeline and tutorials.

.. dropdown:: No enough VRAM or OOM error during evaluation?
:animate: fade-in-slide-down

The default values of :code:`embedder_batch_size` and :code:`reranker_batch_size` are both 3000. Try a smaller value.

Source Introduction IR (`docs/source/Introduction/IR.rst`) Copy Section

Information Retrieval

What is Information Retrieval? Copy Chapter

Simply put, Information Retrieval (IR) is the science of searching and retrieving information from a large collection of data based on a user's query.
The goal of an IR system is not just to return a list of documents but to ensure that the most relevant ones appear at the top of the results.

A very straightforward example of IR is library catalog. One wants to find the book that best matches the query, but there are thousands or millions of books on the shelf.
The library's catalog system helps you find the best matches based on your search terms.
In modern digital world, search engines and databases work in a similar way, using sophisticated algorithms and models to retrieve, rank and return the most relevant results.
And the resource categories are expanding from text to more modalities such as images, videos, 3D objects, music, etc.

IR and Embedding Model Copy Chapter

Traditional IR methods, like TF-IDF and BM25, rely on statistical and heuristic techniques to rank documents based on term frequency and document relevance.
These methods are efficient and effective for keyword-based search but often struggle with understanding the deeper context or semantics of the text.

.. seealso::

Take a very simple example with two sentences:

.. code:: python

    sentence_1 = "watch a play"
    sentence_2 = "play with a watch"

Sentence 1 means going for a show/performance, which has watch as a verb and play as a noun.

However sentence 2 means someone is interacting with a timepiece on wrist, which has play as a verb and watch as a noun.

These two sentences could be regard as very similar to each other when using the traditional IR methods though they actually have totally different semantic meaning.
Then how could we solve this? The best answer up until now is embedding models.

Embedding models have revolutionized IR by representing text as dense vectors in a high-dimensional space, capturing the semantic meaning of words, sentences, or even entire documents.
This allows for more sophisticated search capabilities, such as semantic search, where results are ranked based on meaning rather than simple keyword matching.

Source Introduction Embedder (`docs/source/Introduction/embedder.rst`) Copy Section

Embedder

.. tip::

If you are already familiar with the concepts, take a look at the :doc:BGE models <../bge/index>!

Embedder, or embedding model, bi-encoder, is a model designed to convert data, usually text, codes, or images, into sparse or dense numerical vectors (embeddings) in a high dimensional vector space.
These embeddings capture the semantic meaning or key features of the input, which enable efficient comparison and analysis.

A very famous demonstration is the example from word2vec <https://arxiv.org/abs/1301.3781>_. It shows how word embeddings capture semantic relationships through vector arithmetic:

.. image:: ../_static/img/word2vec.png
:width: 500
:align: center

Nowadays, embedders are capable of mapping sentences and even passages into vector space.
They are widely used in real world tasks such as retrieval, clustering, etc.
In the era of LLMs, embedding models play a pivot role in RAG, enables LLMs to access and integrate relevant context from vast external datasets.

Sparse Vector Copy Chapter

Sparse vectors usually have structure of high dimensionality with only a few non-zero values, which usually effective for tasks like keyword matching.
Typically, though not always, the number of dimensions in sparse vectors corresponds to the different tokens present in the language.
Each dimension is assigned a value representing the token's relative importance within the document.
Some well known algorithms for sparse vector embedding includes bag-of-words <https://en.wikipedia.org/wiki/Bag-of-words_model>, TF-IDF <https://en.wikipedia.org/wiki/Tf%E2%80%93idf>, BM25 <https://en.wikipedia.org/wiki/Okapi_BM25>_, etc.
Sparse vector embeddings have great ability to extract the information of key terms and their corresponding importance within documents.

Dense Vector Copy Chapter

Dense vectors typically use neural networks to map words, sentences, and passages into a fixed dimension latent vector space.
Then we can compare the similarity between two objects using certain metrics like Euclidean distance or Cos similarity.
Those vectors can represent deeper meaning of the sentences.
Thus we can distinguish sentences using similar words but actually have different meaning.
And also understand different ways in speaking and writing that express the same thing.
Dense vector embeddings, instead of keywords counting and matching, directly capture the semantics.

Source Introduction Index (`docs/source/Introduction/index.rst`) Copy Section

Introduction

BGE builds one-stop retrieval toolkit for search and RAG. We provide inference, evaluation, and fine-tuning for embedding models and reranker.

.. figure:: ../_static/img/RAG_pipeline.png
:width: 700
:align: center

BGE embedder and reranker in an RAG pipeline. Source <https://safjan.com/images/retrieval_augmented_generation/RAG.png>_

Quickly get started with:

.. toctree::
:maxdepth: 1
:caption: Start

overview
installation
quick_start

.. toctree::
:maxdepth: 1
:caption: Concept

IR
embedder
reranker
similarity
retrieval_demo

Source Introduction Installation (`docs/source/Introduction/installation.rst`) Copy Section

:github_url: https://github.com/AI4Finance-Foundation/FinRL

Installation

Using pip: Copy Chapter

If you do not need to finetune the models, you can install the package without the finetune dependency:

.. code:: bash

pip install -U FlagEmbedding

If you want to finetune the models, you can install the package with the finetune dependency:

.. code:: bash

pip install -U FlagEmbedding[finetune]

Install from sources: Copy Chapter

Clone the repository and install

.. code:: bash

git clone https://github.com/FlagOpen/FlagEmbedding.git
cd FlagEmbedding
# If you do not need to finetune the models, you can install the package without the finetune dependency:
pip install  .
# If you want to finetune the models, install the package with the finetune dependency:
pip install  .[finetune]

For development in editable mode:

.. code:: bash

# If you do not need to finetune the models, you can install the package without the finetune dependency:
pip install -e .
# If you want to finetune the models, install the package with the finetune dependency:
pip install -e .[finetune]

PyTorch-CUDA Copy Chapter

If you want to use CUDA GPUs during inference and finetuning, please install appropriate version of PyTorch <https://pytorch.org/get-started/locally/>_ with CUDA support.

Source Introduction Overview (`docs/source/Introduction/overview.rst`) Copy Section

Overview

Our repository provides well-structured APIs <https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding>_ for the inference, evaluation, and fine-tuning of BGE series models.
Besides that, there are abundant resources of and for users to quickly get a hands-on experience.

.. figure:: https://raw.githubusercontent.com/FlagOpen/FlagEmbedding/refs/heads/master/imgs/projects.png
:width: 700
:align: center

Structure of contents in our repo <https://github.com/FlagOpen/FlagEmbedding>_

Our repository provides well-structured contents for information retrieval and RAG:

• The core APIs <../API>_ for embedding models' inference, evaluation, and fine-tuning.
• Hands-on examples <https://github.com/FlagOpen/FlagEmbedding/tree/master/examples>_ for the three mentioned use cases.
• Detailed tutorials <https://github.com/FlagOpen/FlagEmbedding/tree/master/Tutorials>_ covering topics in retrieval to help you learn from scratch.

Source Introduction Quick Start (`docs/source/Introduction/quick_start.rst`) Copy Section

Quick Start

First, load one of the BGE embedding model:

.. code:: python

from FlagEmbedding import FlagAutoModel

model = FlagAutoModel.from_finetuned('BAAI/bge-base-en-v1.5')

.. tip::

If there's difficulty connecting to Hugging Face, you can use the `HF mirror <https://hf-mirror.com/>`_ instead.

.. code:: bash

    export HF_ENDPOINT=https://hf-mirror.com

Then, feed some sentences to the model and get their embeddings:

.. code:: python

sentences_1 = ["I love NLP", "I love machine learning"]
sentences_2 = ["I love BGE", "I love text retrieval"]
embeddings_1 = model.encode(sentences_1)
embeddings_2 = model.encode(sentences_2)

Once we get the embeddings, we can compute similarity by inner product:

.. code:: python

similarity = embeddings_1 @ embeddings_2.T
print(similarity)

Source Introduction Reranker (`docs/source/Introduction/reranker.rst`) Copy Section

Reranker

.. tip::

If you are already familiar with the concepts, take a look at the :doc:BGE rerankers <../bge/index>!

Reranker, or Cross-Encoder, is a model that refines the ranking of candidate pairs (e.g., query-document pairs) by jointly encoding and scoring them.

Typically, we use embedder as a Bi-Encoder. It first computes the embeddings of two input sentences, then compute their similarity using metrics such as cosine similarity or Euclidean distance.
Whereas a reranker takes two sentences at the same time and directly computer a score representing their similarity.

The following figure shows their difference:

.. figure:: https://raw.githubusercontent.com/UKPLab/sentence-transformers/master/docs/img/Bi_vs_Cross-Encoder.png
:width: 500
:align: center

Bi-Encoder & Cross-Encoder (from Sentence Transformers)

Although Cross-Encoder usually has better performances than Bi-Encoder, it is extremly time consuming to use Cross-Encoder if we have a great amount of data.
Thus a widely accepted approach is to use a Bi-Encoder for initial retrieval (e.g., selecting the top 100 candidates from 100,000 sentences) and then refine the ranking of the selected candidates using a Cross-Encoder for more accurate results.

Source Introduction Similarity (`docs/source/Introduction/similarity.rst`) Copy Section

Similarity

A primary goal of retrieval is to find the most relevant documents in response to a user's query.
One of the core components of this process is measuring similarity between the query and candidates.
Similarity metrics quantify how closely related two pieces of data are, and guide the retrieval system in ranking results.

Jaccard Similarity Copy Chapter

.. math::

J(A,B)=\frac{|A\cap B|}{|A\cup B|}

The Jaccard similarity or Jaccard index is commonly used for set-based similarity, particularly in binary data (e.g., whether a term appears in a document or not).
It is calculated as the size of the intersection of two sets divided by the size of their union.
In information retrieval, it's often used to compare sets of keywords or phrases, with higher values indicating more similarity.

Euclidean Distance Copy Chapter

.. math::

d(A, B) = \|A-B\|_2 = \sqrt{\sum_{i=1}^n (A_i-B_i)^2}

Euclidean distance measures the straight-line distance between two points in a vector space.
In IR, this can be used to assess the difference between document or query vectors.
A smaller distance indicates greater similarity.
This metric is intuitive but can sometimes be sensitive to the scale of the data, especially in high-dimensional spaces like text embeddings.

Cosine Similarity Copy Chapter

.. math::

\cos(\theta)=\frac{A\cdot B}{\|A\|\|B\|}

Cosine similarity is one of the most widely used metrics in information retrieval, especially for text.
It measures the cosine of the angle between two vectors in a multi-dimensional space (typically representing term frequency vectors of documents and queries).
If the cosine similarity is closer to 1, the vectors are more similar.
A value of 0 indicates orthogonality, meaning no similarity.
It's a simple yet effective measure for text-based retrieval, as it considers the orientation but not the magnitude of vectors.

Dot Product Copy Chapter

Coordinate definition:
.. math::

A\cdot B = \sum_{i=1}^{i=n}A_i B_i

Geometric definition:
.. math::

A\cdot B = \|A\|\|B\|\cos(\theta)

The dot product between two vectors provides a measure of how similar the vectors are in terms of direction and magnitude.
In information retrieval, the dot product is often used in vector space models, particularly when dealing with pre-trained word or sentence embeddings.
A higher dot product indicates that the query and document are closely aligned in the vector space.

Source Bge Bge Code (`docs/source/bge/bge_code.rst`) Copy Section

BGE-Code-v1

BGE-Code-v1 <https://huggingface.co/BAAI/bge-code-v1>_ is an LLM-based code embedding model that supports code retrieval, text retrieval, and multilingual retrieval. It primarily demonstrates the following capabilities:

• Superior Code Retrieval Performance: The model demonstrates exceptional code retrieval capabilities, supporting natural language queries in both English and Chinese, as well as 20 programming languages.
• Robust Text Retrieval Capabilities: The model maintains strong text retrieval capabilities comparable to text embedding models of similar scale.
• Extensive Multilingual Support: BGE-Code-v1 offers comprehensive multilingual retrieval capabilities, excelling in languages such as English, Chinese, Japanese, French, and more.

+-------------------------------------------------------------------+-----------------+------------+--------------+----------------------------------------------------------------------------------------------------+
| Model | Language | Parameters | Model Size | Description |
+===================================================================+=================+============+==============+====================================================================================================+
| BAAI/bge-code-v1 <https://huggingface.co/BAAI/bge-code-v1>_ | Multilingual | 1.5B | 6.18 GB | SOTA code retrieval model, with exceptional multilingual text retrieval performance as well |
+-------------------------------------------------------------------+-----------------+------------+--------------+----------------------------------------------------------------------------------------------------+

.. code:: python
from FlagEmbedding import FlagLLMModel

queries = [
    "Delete the record with ID 4 from the 'Staff' table.", 
    'Delete all records in the "Livestock" table where age is greater than 5'
]
documents = [
    "DELETE FROM Staff WHERE StaffID = 4;",
    "DELETE FROM Livestock WHERE age > 5;"
]

model = FlagLLMModel('BAAI/bge-code-v1', 
                    query_instruction_format="{}\n<query>{}",
                    query_instruction_for_retrieval="Given a question in text, retrieve SQL queries that are appropriate responses to the question.",
                    trust_remote_code=True,
                    use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
embeddings_1 = model.encode_queries(queries)
embeddings_2 = model.encode_corpus(documents)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)

Source Bge Bge Icl (`docs/source/bge/bge_icl.rst`) Copy Section

BGE-EN-ICL

BGE-EN-ICL is the new SoTA embedding model in BGE series with capabilities:

• In-context learning ability: By providing few-shot examples in the query, it can significantly enhance the model's ability to handle new tasks.
• Outstanding performance: The model has achieved state-of-the-art (SOTA) performance on MTEB and AIR-Bench.

+-------------------------------------------------------------------+-----------------+------------+--------------+----------------------------------------------------------------------------------------------------+
| Model | Language | Parameters | Model Size | Description |
+===================================================================+=================+============+==============+====================================================================================================+
| BAAI/bge-en-icl <https://huggingface.co/BAAI/bge-en-icl>_ | English | 7.1B | 28.5 GB | In-context learning capabilities, fully leverage the model's potential based on a few shot examples|
+-------------------------------------------------------------------+-----------------+------------+--------------+----------------------------------------------------------------------------------------------------+

Usage Copy Chapter

.. code:: python

from FlagEmbedding import FlagICLModel

documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "Definition of summit for English Language Learners. : 1  the highest point of a mountain : the top of a mountain. : 2  the highest level. : 3  a meeting or series of meetings between the leaders of two or more governments."
]

examples = [
    {
        'instruct': 'Given a web search query, retrieve relevant passages that answer the query.',
        'query': 'what is a virtual interface',
        'response': "A virtual interface is a software-defined abstraction that mimics the behavior and characteristics of a physical network interface. It allows multiple logical network connections to share the same physical network interface, enabling efficient utilization of network resources. Virtual interfaces are commonly used in virtualization technologies such as virtual machines and containers to provide network connectivity without requiring dedicated hardware. They facilitate flexible network configurations and help in isolating network traffic for security and management purposes."
    },
    {
        'instruct': 'Given a web search query, retrieve relevant passages that answer the query.',
        'query': 'causes of back pain in female for a week',
        'response': "Back pain in females lasting a week can stem from various factors. Common causes include muscle strain due to lifting heavy objects or improper posture, spinal issues like herniated discs or osteoporosis, menstrual cramps causing referred pain, urinary tract infections, or pelvic inflammatory disease. Pregnancy-related changes can also contribute. Stress and lack of physical activity may exacerbate symptoms. Proper diagnosis by a healthcare professional is crucial for effective treatment and management."
    }
]

queries = ["how much protein should a female eat", "summit define"]

model = FlagICLModel('BAAI/bge-en-icl', 
                     examples_for_task=examples,  # set `examples_for_task=None` to use model without examples
                     examples_instruction_format="{}\n<query>{}\n<response>{}") # specify the format to use examples_for_task

embeddings_1 = model.encode_queries(queries)
embeddings_2 = model.encode_corpus(documents)
similarity = embeddings_1 @ embeddings_2.T

print(similarity)

Source Bge Bge M3 (`docs/source/bge/bge_m3.rst`) Copy Section

====== BGE-M3

BGE-M3 is a compound and powerful embedding model distinguished for its versatility in:

• Multi-Functionality: It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval.
• Multi-Linguality: It can support more than 100 working languages.
• Multi-Granularity: It is able to process inputs of different granularities, spanning from short sentences to long documents of up to 8192 tokens.

+-------------------------------------------------------------------+-----------------+------------+--------------+-----------------------------------------------------------------------+
| Model | Language | Parameters | Model Size | Description |
+===================================================================+=================+============+==============+=======================================================================+
| BAAI/bge-m3 <https://huggingface.co/BAAI/bge-m3>_ | Multi-Lingual | 569M | 2.27 GB | Multi-Functionality, Multi-Linguality, and Multi-Granularity |
+-------------------------------------------------------------------+-----------------+------------+--------------+-----------------------------------------------------------------------+

Multi-Linguality

BGE-M3 was trained on multiple datasets covering up to 170+ different languages.
While the amount of training data on languages are highly unbalanced, the actual model performance on different languages will have difference.

For more information of datasets and evaluation results, please check out our paper <https://arxiv.org/pdf/2402.03216s>_ for details.

Multi-Granularity

We extend the max position to 8192, enabling the embedding of larger corpus.
Proposing a simple but effective method: MCLS (Multiple CLS) to enhance the model's ability on long text without additional fine-tuning.

Multi-Functionality

.. code:: python

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel('BAAI/bge-m3')
sentences_1 = ["What is BGE M3?", "Defination of BM25"]
sentences_2 = ["BGE M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.", 
               "BM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document"]

Dense Retrieval Copy Chapter

Similar to BGE v1 or v1.5 models, BGE-M3 use the normalized hidden state of the special token [CLS] as the dense embedding:

.. math:: e_q = norm(H_q[0])

Next, to compute the relevance score between the query and passage:

.. math:: s_{dense}=f_{sim}(e_p, e_q)

where :math:e_p, e_q are the embedding vectors of passage and query, respectively.

:math:f_{sim} is the score function (such as inner product and L2 distance) for comupting two embeddings' similarity.

Sparse Retrieval Copy Chapter

BGE-M3 generates sparce embeddings by adding a linear layer and a ReLU activation function following the hidden states:

.. math:: w_{qt} = \text{Relu}(W_{lex}^T H_q [i])

where :math:W_{lex} representes the weights of linear layer and :math:H_q[i] is the encoder's output of the :math:i^{th} token.

Based on the tokens' weights of query and passage, the relevance score between them is computed by the joint importance of the co-existed terms within the query and passage:

.. math:: s_{lex} = \sum_{t\in q\cap p}(w_{qt} * w_{pt})

where :math:w_{qt}, w_{pt} are the importance weights of each co-existed term :math:t in query and passage, respectively.

Multi-Vector Copy Chapter

The multi-vector method utilizes the entire output embeddings for the representation of query :math:E_q and passage :math:E_p.

.. math::

E_q = norm(W_{mul}^T H_q)

E_p = norm(W_{mul}^T H_p)

where :math:W_{mul} is the learnable projection matrix.

Following ColBert, BGE-M3 use late-interaction to compute the fine-grained relevance score:

.. math:: s_{mul}=\frac{1}{N}\sum_{i=1}^N\max_{j=1}^M E_q[i]\cdot E_p^T[j]

where :math:E_q, E_p are the entire output embeddings of query and passage, respectively.

This is a summation of average of maximum similarity of each :math:v\in E_q with vectors in :math:E_p.

Hybrid Ranking Copy Chapter

BGE-M3's multi-functionality gives the possibility of hybrid ranking to improve retrieval.
Firstly, due to the heavy cost of multi-vector method, we can retrieve the candidate results by either of the dense or sparse method.
Then, to get the final result, we can rerank the candidates based on the integrated relevance score:

.. math:: s_{rank} = w_1\cdot s_{dense}+w_2\cdot s_{lex} + w_3\cdot s_{mul}

where the values chosen for :math:w_1, :math:w_2 and :math:w_3 varies depending on the downstream scenario.

Usage

.. code:: python

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel('BAAI/bge-m3')

sentences_1 = ["What is BGE M3?", "Defination of BM25"]

output = model.encode(sentences_1, return_dense=True, return_sparse=True, return_colbert_vecs=True)
dense, sparse, multiv = output['dense_vecs'], output['lexical_weights'], output['colbert_vecs']

Useful Links:

API <../API/inference/embedder/encoder_only/M3Embedder>_
Tutorial <https://github.com/FlagOpen/FlagEmbedding/blob/master/Tutorials/1_Embedding/1.2.4_BGE-M3.ipynb>_
Example <https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/inference/embedder/encoder_only>_

Source Bge Bge Reranker (`docs/source/bge/bge_reranker.rst`) Copy Section

BGE-Reranker

Different from embedding model, reranker, or cross-encoder uses question and document as input and directly output similarity instead of embedding.
To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models.
For examples, use a bge embedding model to first retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.

The first series of BGE-Reranker contains two models, large and base.

+-------------------------------------------------------------------------------+-----------------------+------------+--------------+-----------------------------------------------------------------------+
| Model | Language | Parameters | Model Size | Description |
+===============================================================================+=======================+============+==============+=======================================================================+
| BAAI/bge-reranker-large <https://huggingface.co/BAAI/bge-reranker-large>_ | English & Chinese | 560M | 2.24 GB | Larger reranker model, easy to deploy with better inference |
+-------------------------------------------------------------------------------+-----------------------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-reranker-base <https://huggingface.co/BAAI/bge-reranker-base>_ | English & Chinese | 278M | 1.11 GB | Lightweight reranker model, easy to deploy with fast inference |
+-------------------------------------------------------------------------------+-----------------------+------------+--------------+-----------------------------------------------------------------------+

bge-reranker-large and bge-reranker-base used XLM-RoBERTa-Large <https://huggingface.co/FacebookAI/xlm-roberta-large>_ and XLM-RoBERTa-Base <https://huggingface.co/FacebookAI/xlm-roberta-base>_ respectively as the base model.
They were trained on high quality English and Chinese data, and acheived State-of-The-Art performance in the level of same size models at the time released.

Usage Copy Chapter

.. code:: python

from FlagEmbedding import FlagReranker

reranker = FlagReranker(
    'BAAI/bge-reranker-base', 
    query_max_length=256,
    use_fp16=True,
    devices=['cuda:1'],
)

score = reranker.compute_score(['I am happy to help', 'Assisting you is my pleasure'])
print(score)

Source Bge Bge Reranker V2 (`docs/source/bge/bge_reranker_v2.rst`) Copy Section

BGE-Reranker-v2

+------------------------------------------------------------------------------------------------------------------+-----------------------+-------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Language | Parameters | Model Size | Description |
+==================================================================================================================+=======================+=============+==============+=========================================================================================================================================================+
| BAAI/bge-reranker-v2-m3 <https://huggingface.co/BAAI/bge-reranker-v2-m3>_ | Multilingual | 568M | 2.27 GB | Lightweight reranker model, possesses strong multilingual capabilities, easy to deploy, with fast inference. |
+------------------------------------------------------------------------------------------------------------------+-----------------------+-------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| BAAI/bge-reranker-v2-gemma <https://huggingface.co/BAAI/bge-reranker-v2-gemma>_ | Multilingual | 2.51B | 10 GB | Suitable for multilingual contexts, performs well in both English proficiency and multilingual capabilities. |
+------------------------------------------------------------------------------------------------------------------+-----------------------+-------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| BAAI/bge-reranker-v2-minicpm-layerwise <https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise>_ | Multilingual | 2.72B | 10.9 GB | Suitable for multilingual contexts, allows freedom to select layers for output, facilitating accelerated inference. |
+------------------------------------------------------------------------------------------------------------------+-----------------------+-------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| BAAI/bge-reranker-v2.5-gemma2-lightweight <https://huggingface.co/BAAI/bge-reranker-v2.5-gemma2-lightweight>_ | Multilingual | 2.72B | 10.9 GB | Suitable for multilingual contexts, allows freedom to select layers, compress ratio and compress layers for output, facilitating accelerated inference. |
+------------------------------------------------------------------------------------------------------------------+-----------------------+-------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+

.. tip::

You can select the model according your senario and resource:

- For multilingual, utilize :code:`BAAI/bge-reranker-v2-m3`, :code:`BAAI/bge-reranker-v2-gemma` and :code:`BAAI/bge-reranker-v2.5-gemma2-lightweight`.
- For Chinese or English, utilize :code:`BAAI/bge-reranker-v2-m3` and :code:`BAAI/bge-reranker-v2-minicpm-layerwise`.
- For efficiency, utilize :code:`BAAI/bge-reranker-v2-m3` and the low layer of :code:`BAAI/bge-reranker-v2-minicpm-layerwise`.
- For better performance, recommand :code:`BAAI/bge-reranker-v2-minicpm-layerwise` and :code:`BAAI/bge-reranker-v2-gemma`.

Make sure always test on your real use case and choose the one with best speed-quality balance!

Usage Copy Chapter

bge-reranker-v2-m3

Use :code:bge-reranker-v2-m3 in the same way as bge-reranker-base and bge-reranker-large.

.. code:: python

from FlagEmbedding import FlagReranker

# Setting use_fp16 to True speeds up computation with a slight performance degradation
reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True)

score = reranker.compute_score(['query', 'passage'])
# or set "normalize=True" to apply a sigmoid function to the score for 0-1 range
score = reranker.compute_score(['query', 'passage'], normalize=True)

print(score)

bge-reranker-v2-gemma

Use the :code:FlagLLMReranker class for bge-reranker-v2-gemma.

.. code:: python

from FlagEmbedding import FlagLLMReranker

# Setting use_fp16 to True speeds up computation with a slight performance degradation
reranker = FlagLLMReranker('BAAI/bge-reranker-v2-gemma', use_fp16=True)

score = reranker.compute_score(['query', 'passage'])
print(score)

bge-reranker-v2-minicpm-layerwise

Use the :code:LayerWiseFlagLLMReranker class for bge-reranker-v2-minicpm-layerwise.

.. code:: python

from FlagEmbedding import LayerWiseFlagLLMReranker

# Setting use_fp16 to True speeds up computation with a slight performance degradation
reranker = LayerWiseFlagLLMReranker('BAAI/bge-reranker-v2-minicpm-layerwise', use_fp16=True)

# Adjusting 'cutoff_layers' to pick which layers are used for computing the score.
score = reranker.compute_score(['query', 'passage'], cutoff_layers=[28]) 
print(score)

bge-reranker-v2.5-gemma2-lightweight

Use the :code:LightWeightFlagLLMReranker class for bge-reranker-v2.5-gemma2-lightweight.

.. code:: python

from FlagEmbedding import LightWeightFlagLLMReranker

# Setting use_fp16 to True speeds up computation with a slight performance degradation
reranker = LightWeightFlagLLMReranker('BAAI/bge-reranker-v2.5-gemma2-lightweight', use_fp16=True)

# Adjusting 'cutoff_layers' to pick which layers are used for computing the score.
score = reranker.compute_score(['query', 'passage'], cutoff_layers=[28], compress_ratio=2, compress_layer=[24, 40])
print(score)

Source Bge Bge V1 V1.5 (`docs/source/bge/bge_v1_v1.5.rst`) Copy Section

BGE v1 & v1.5

BGE v1 and v1.5 are series of encoder only models base on BERT. They achieved best performance among the models of the same size at the time of release.

BGE Copy Chapter

The first group of BGE models was released in Aug 2023. The :code:bge-large-en and :code:bge-large-zh ranked 1st on MTEB and
C-MTEB benchmarks at the time released.

+-------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| Model | Language | Parameters | Model Size | Description |
+===================================================================+===========+============+==============+=======================================================================+
| BAAI/bge-large-en <https://huggingface.co/BAAI/bge-large-en>_ | English | 335M | 1.34 GB | Embedding Model which map text into vector |
+-------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-base-en <https://huggingface.co/BAAI/bge-base-en>_ | English | 109M | 438 MB | a base-scale model but with similar ability to BAAI/bge-large-en |
+-------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-small-en <https://huggingface.co/BAAI/bge-small-en>_ | English | 33.4M | 133 MB | a small-scale model but with competitive performance |
+-------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-large-zh <https://huggingface.co/BAAI/bge-large-zh>_ | Chinese | 326M | 1.3 GB | Embedding Model which map text into vector |
+-------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-base-zh <https://huggingface.co/BAAI/bge-base-zh>_ | Chinese | 102M | 409 MB | a base-scale model but with similar ability to BAAI/bge-large-zh |
+-------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-small-zh <https://huggingface.co/BAAI/bge-small-zh>_ | Chinese | 24M | 95.8 MB | a small-scale model but with competitive performance |
+-------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+

BGE-v1.5 Copy Chapter

Then to enhance its retrieval ability without instruction and alleviate the issue of the similarity distribution, :code:bge-*-v1.5 models
were released in Sep 2023. They are still the most popular embedding models that balanced well between embedding quality and model sizes.

+-----------------------------------------------------------------------------+-----------+------------+--------------+--------------+
| Model | Language | Parameters | Model Size | Description |
+=============================================================================+===========+============+==============+==============+
| BAAI/bge-large-en-v1.5 <https://huggingface.co/BAAI/bge-large-en-v1.5>_ | English | 335M | 1.34 GB | version 1.5 |
+-----------------------------------------------------------------------------+-----------+------------+--------------+ with more +
| BAAI/bge-base-en-v1.5 <https://huggingface.co/BAAI/bge-base-en-v1.5>_ | English | 109M | 438 MB | reasonable |
+-----------------------------------------------------------------------------+-----------+------------+--------------+ similarity +
| BAAI/bge-small-en-v1.5 <https://huggingface.co/BAAI/bge-small-en-v1.5>_ | English | 33.4M | 133 MB | distribution |
+-----------------------------------------------------------------------------+-----------+------------+--------------+ and better +
| BAAI/bge-large-zh-v1.5 <https://huggingface.co/BAAI/bge-large-zh-v1.5>_ | Chinese | 326M | 1.3 GB | performance |
+-----------------------------------------------------------------------------+-----------+------------+--------------+ +
| BAAI/bge-base-zh-v1.5 <https://huggingface.co/BAAI/bge-base-zh-v1.5>_ | Chinese | 102M | 409 MB | |
+-----------------------------------------------------------------------------+-----------+------------+--------------+ +
| BAAI/bge-small-zh-v1.5 <https://huggingface.co/BAAI/bge-small-zh-v1.5>_ | Chinese | 24M | 95.8 MB | |
+-----------------------------------------------------------------------------+-----------+------------+--------------+--------------+

Usage Copy Chapter

To use BGE v1 or v1.5 model for inference, load model through

.. code:: python

from FlagEmbedding import FlagModel

model = FlagModel('BAAI/bge-base-en-v1.5')

sentences = ["Hello world", "I am inevitable"]
embeddings = model.encode(sentences)

.. tip::

For simple tasks that only encode a few sentences like above, it's faster to use CPU or a single GPU instead of multi-GPUs

To use CPU:

.. code:: python

# make no GPU visible
import os
os.environ['CUDA_VISIBLE_DEVICES'] = ''

# or claim the devices during initialize the model
model = FlagModel('BAAI/bge-base-en-v1.5', devices='cpu')

To use a single GPU:

.. code:: python

# select one sigle card to be visible
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

# or claim the devices during initialize the model
model = FlagModel('BAAI/bge-base-en-v1.5', devices=0)

|

Useful Links:

API <../API/inference/embedder/encoder_only/BaseEmbedder>_

Tutorial <https://github.com/FlagOpen/FlagEmbedding/blob/master/Tutorials/1_Embedding/1.2.3_BGE_v1%261.5.ipynb>_

Example <https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/inference/embedder/encoder_only>_

Source Bge Bge Vl (`docs/source/bge/bge_vl.rst`) Copy Section

BGE-VL

BGE-VL is a series of multimodel retrieval models training on MegaPairs <https://github.com/VectorSpaceLab/MegaPairs>_

BGE-VL contains light weight CLIP based models as well as more powerful LLAVA-NeXT based MLLM models:

+----------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| Model | Language | Parameters | Model Size | Description |
+======================================================================+===========+============+==============+=======================================================================+
| BAAI/bge-vl-base <https://huggingface.co/BAAI/BGE-VL-base>_ | English | 150M | 299 MB | Light weight multimodel embedder among image and text |
+----------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-vl-large <https://huggingface.co/BAAI/BGE-VL-large>_ | English | 428M | 855 MB | Large scale multimodel embedder among image and text |
+----------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-vl-MLLM-S1 <https://huggingface.co/BAAI/BGE-VL-MLLM-S1>_ | English | 7.57B | 15.14 GB | SOTA in composed image retrieval, trained on MegaPairs dataset |
+----------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/bge-vl-MLLM-S2 <https://huggingface.co/BAAI/BGE-VL-MLLM-S2>_ | English | 7.57B | 15.14 GB | Finetune BGE-VL-MLLM-S1 with one epoch on MMEB training set |
+----------------------------------------------------------------------+-----------+------------+--------------+-----------------------------------------------------------------------+
| BAAI/BGE-VL-v1.5-zs <https://huggingface.co/BAAI/BGE-VL-v1.5-zs>_ | English | 7.57B | 15.14 GB | Better multi-modal retrieval model with performs well in all kinds of tasks |
| BAAI/BGE-VL-v1.5-mmeb <https://huggingface.co/BAAI/BGE-VL-v1.5-mmeb>_ | English | 7.57B | 15.14 GB | Better multi-modal retrieval model, additionally fine-tuned on MMEB training set |

BGE-VL-CLIP Copy Chapter

The base and large model are trained based on CLIP-vit-base-patch16 and CLIP-vit-large-patch14.
For composed image-text data, the model directly use score-fusion to sum up the outputs of visual encoder and text encoder and get the final embedding.

.. tip::

Our code works well on transformers==4.45.2, and we recommend using this version.

You can easily use BGE-VL-CLIP models based on transformers:

.. code:: python

import torch
from transformers import AutoModel

MODEL_NAME = "BAAI/BGE-VL-base" # or "BAAI/BGE-VL-large"
model = AutoModel.from_pretrained(MODEL_NAME, trust_remote_code=True) # You must set trust_remote_code=True
model.set_processor(MODEL_NAME)
model.eval()

with torch.no_grad():
    query = model.encode(
        images = "./assets/cir_query.png", 
        text = "Make the background dark, as if the camera has taken the photo at night"
    )
    candidates = model.encode(
        images = ["./assets/cir_candi_1.png", "./assets/cir_candi_2.png"]
    )
    
    scores = query @ candidates.T
print(scores)

BGE-VL-MLLM Copy Chapter

The multimodal large language models (MLLMs) incorporate a visual encoder, typically based on a vision transformer, into a large language model (LLM).
This integration allows image tokens to be directly processed by the LLM.
Consequently, MLLMs can effectively handle diverse multimodal inputs by converting any type of input into a sequence of tokens.

BGE-VL-MLLM builds upon the LLaVA1.6. In both training and inference stages, MMRet uses task-specific instructions for query inputs to improve generalization, aligning
with standard practices in LLM-based embedding models.
A typical multimodal query input is structured as follows:

.. math::

⟨\text{instruct}⟩{\{task\_ inst\}} \space⟨\text{query}⟩\{q_t\} \{q_i\}\space[\text{EOS}]

where :math:{task_inst} represents the task-specific instruction, :math:{qt} denotes the input query text, and
:math:{qi} is the input query image.
The normalized last hidden state of the [EOS] token in the MLLM is used as the embedding of any given input sequence.

.. code:: python

import torch
from transformers import AutoModel
from PIL import Image

MODEL_NAME= "BAAI/BGE-VL-MLLM-S1"
model = AutoModel.from_pretrained(MODEL_NAME, trust_remote_code=True)
model.eval()
model.cuda()

with torch.no_grad():
    model.set_processor(MODEL_NAME)

    query_inputs = model.data_process(
        text="Make the background dark, as if the camera has taken the photo at night", 
        images="./assets/cir_query.png",
        q_or_c="q",
        task_instruction="Retrieve the target image that best meets the combined criteria by using both the provided image and the image retrieval instructions: "
    )
    candidate_inputs = model.data_process(
        images=["./assets/cir_candi_1.png", "./assets/cir_candi_2.png"],
        q_or_c="c",
    )

    query_embs = model(**query_inputs, output_hidden_states=True)[:, -1, :]
    candi_embs = model(**candidate_inputs, output_hidden_states=True)[:, -1, :]
    
    query_embs = torch.nn.functional.normalize(query_embs, dim=-1)
    candi_embs = torch.nn.functional.normalize(candi_embs, dim=-1)

    scores = torch.matmul(query_embs, candi_embs.T)
print(scores)

BGE-VL-v1.5 Copy Chapter

BGE-VL-v1.5 series is the updated version of BGE-VL, bringing better performance on both retrieval and multi-modal understanding. The models were trained on 30M MegaPairs data and extra 10M natural and synthetic data.

bge-vl-v1.5-zs is a zero-shot model, only trained on the data mentioned above. bge-vl-v1.5-mmeb is the fine-tuned version on MMEB training set.

.. code:: python

import torch
from transformers import AutoModel
from PIL import Image

MODEL_NAME= "BAAI/BGE-VL-v1.5-mmeb" # "BAAI/BGE-VL-v1.5-zs"

model = AutoModel.from_pretrained(MODEL_NAME, trust_remote_code=True)
model.eval()
model.cuda()

with torch.no_grad():
    model.set_processor(MODEL_NAME)

    query_inputs = model.data_process(
        text="Make the background dark, as if the camera has taken the photo at night", 
        images="../../imgs/cir_query.png",
        q_or_c="q",
        task_instruction="Retrieve the target image that best meets the combined criteria by using both the provided image and the image retrieval instructions: "
    )

    candidate_inputs = model.data_process(
        images=["../../imgs/cir_candi_1.png", "../../imgs/cir_candi_2.png"],
        q_or_c="c",
    )

    query_embs = model(**query_inputs, output_hidden_states=True)[:, -1, :]
    candi_embs = model(**candidate_inputs, output_hidden_states=True)[:, -1, :]
    
    query_embs = torch.nn.functional.normalize(query_embs, dim=-1)
    candi_embs = torch.nn.functional.normalize(candi_embs, dim=-1)

    scores = torch.matmul(query_embs, candi_embs.T)
print(scores)

For more details, check out the repo of MegaPairs <https://github.com/VectorSpaceLab/MegaPairs>_

Source Bge Index (`docs/source/bge/index.rst`) Copy Section

BGE

.. figure:: ../_static/img/bge_logo.jpeg
:width: 250
:align: center

BGE stands for BAAI General Embeddings, which is a series of embedding models released by BAAI.

.. toctree::
:maxdepth: 1
:caption: Embedder

bge_v1_v1.5
bge_m3
bge_icl
bge_vl
bge_code

.. toctree::
:maxdepth: 1
:caption: Reranker

bge_reranker
bge_reranker_v2

Source Community Index (`docs/source/community/index.rst`) Copy Section

Community

Visit our GitHub repo <https://github.com/FlagOpen/FlagEmbedding>_ and
Hugging Face collection <https://huggingface.co/collections/BAAI/bge-66797a74476eb1f085c7446d>_ for more materials!

We are also holding WeChat groups for for BGE. Scan the QR code to join the group chat!
To get the first hand message about our updates and new release, or having any questions or ideas, join us now!

.. figure:: ../_static/img/BGE_WeChat_Group.png
:width: 400
:align: center

Source Tutorial 1 Embedding (`docs/source/tutorial/1_Embedding.rst`) Copy Section

1. Embedding

.. toctree::
:hidden:
:maxdepth: 1
:caption: Embedding

1_Embedding/1.1.1
1_Embedding/1.2.1
1_Embedding/1.2.2
1_Embedding/1.2.3
1_Embedding/1.2.4
1_Embedding/1.2.5

Source Tutorial 2 Metrics (`docs/source/tutorial/2_Metrics.rst`) Copy Section

2. Metrics

.. toctree::
:hidden:
:maxdepth: 1
:caption: Metrics

2_Metrics/2.1
2_Metrics/2.2

Source Tutorial 3 Indexing (`docs/source/tutorial/3_Indexing.rst`) Copy Section

3. Indexing

.. toctree::
:hidden:
:maxdepth: 1
:caption: Indexing

3_Indexing/3.1.1
3_Indexing/3.1.2
3_Indexing/3.1.3
3_Indexing/3.1.4
3_Indexing/3.1.5

Source Tutorial 4 Evaluation (`docs/source/tutorial/4_Evaluation.rst`) Copy Section

4. Evaluation

.. toctree::
:hidden:
:maxdepth: 1
:caption: Evaluation

4_Evaluation/4.1.1
4_Evaluation/4.2.1
4_Evaluation/4.2.2
4_Evaluation/4.2.3
4_Evaluation/4.3.1
4_Evaluation/4.4.1
4_Evaluation/4.5.1
4_Evaluation/4.5.2

Source Tutorial 5 Reranking (`docs/source/tutorial/5_Reranking.rst`) Copy Section

5. Reranking

.. toctree::
:hidden:
:maxdepth: 1
:caption: Reranking

5_Reranking/5.1
5_Reranking/5.2
5_Reranking/5.3

Source Tutorial 6 RAG (`docs/source/tutorial/6_RAG.rst`) Copy Section

6. RAG

.. toctree::
:hidden:
:maxdepth: 1
:caption: RAG

6_RAG/6.1
6_RAG/6.2
6_RAG/6.3

Source Tutorial 7 Finetuning (`docs/source/tutorial/7_Finetuning.rst`) Copy Section

7. Finetuning

.. toctree::
:hidden:
:maxdepth: 1
:caption: Finetuning

7_Finetuning/7.1.1
7_Finetuning/7.1.2
7_Finetuning/7.1.3
7_Finetuning/7.2.1

Source Tutorial Index (`docs/source/tutorial/index.rst`) Copy Section

Tutorials

In this section, we provide hands on introduction to different topics that highly related to embedding models and retrieval.

To run the tutorials, clone the GitHub repo and check the Tutorials <https://github.com/FlagOpen/FlagEmbedding/tree/master/Tutorials>_ folder.

.. toctree::
:maxdepth: 1
:caption: Tutorials

1_Embedding
2_Metrics
3_Indexing
4_Evaluation
5_Reranking
6_RAG
7_Finetuning

Research Llm Embedder Docs Evaluation (`research/llm_embedder/docs/evaluation.md`) Copy Section

Evaluation

LLM-Embedder supports 6 retrieval-augmentation tasks tailored for modern LLMs, including:

• Question Answering (qa)
  • evaluate with eval_popqa and eval_mmlu
• In-Context Learning (icl)
  • evaluate with eval_icl
• Long Conversation (chat)
  • evaluate with eval_msc
• Long-Range Language Modeling (lrlm)
  • evaluate with eval_lrlm
• Tool Learning (tool)
  • evaluate with eval_tool
• Conversational Search (convsearch)
  • evaluate with eval_qrecc

Environment Copy Chapter

It is recommended that you create a new environment:

cd FlagEmbedding/llm_embedder

conda env create -f environment.yaml --name llm-embedder
conda activate llm-embedder

To use BM25, you must download java11 and anserini, then add java to your PATH:

# feel free to alternate /data to your prefered location
wget https://huggingface.co/datasets/namespace-Pt/projects/resolve/main/java11.tar.gz?download=true -O /data/java11.tar.gz
wget https://huggingface.co/datasets/namespace-Pt/projects/resolve/main/anserini.tar.gz?download=true -O /data/anserini.tar.gz

cd /data
tar -xzvf java11.tar.gz
tar -xzvf anserini.tar.gz

# below just temporarily set JAVA_HOME; it is RECOMMENDED that you store the lines the setting in ~/.bashrc
export JAVA_HOME=/data/jdk-11.0.2
export PATH=$JAVA_HOME/bin:$PATH

Data Copy Chapter

You should download the data for fine-tuning & evaluation then untar the file at anywhere you prefer, e.g. /data, which results in a folder /data/llm-embedder:

# feel free to alternate /data to your prefered location
wget https://huggingface.co/datasets/namespace-Pt/projects/resolve/main/llm-embedder.tar.gz?download=true -O /data/llm-embedder.tar.gz

cd /data
tar -xzvf llm-embedder-eval.tar.gz

The corpus of QReCC for conversational search is too large (54M passages), we separately upload it to huggingface datasets namespace-Pt/qrecc-corpus. To evaluate the performance on conversational search, you should load it and save it as json file in the qrecc folder:

import datasets
# load dataset
qrecc_corpus = datasets.load_dataset("namespace-Pt/qrecc-corpus", split="train")
# save to jsonline format in YOUR data folder
qrecc_corpus.to_json("/data/llm-embedder/convsearch/qrecc/corpus.json", force_ascii=False, lines=True, orient="records")

Benchmark Copy Chapter

Commands Copy Section

Below are commands to run evaluation for different retrieval models. You can replace eval_popqa with any of eval_mmlu, eval_icl, eval_lrlm, eval_msc, eval_tool, and eval_qrecc. The results will be logged at data/results/.

All our evaluation are based on meta-llama/Llama-2-7b-chat-hf. To use different language models, e.g. Qwen/Qwen-7B-Chat, simply add --model_name_or_path Qwen/Qwen-7B-Chat after every command.

Note that you can modify the default value of data_root in src/retrieval/args.py, so that you don't need to type it for each command.

/* Detailed source-code truncated for AI context efficiency. */

For Instructor, we should first convert it to our format:

# convert sentence transformer based Instructor to our format
import torch
from src.retrieval import DenseRetriever, RetrievalArgs
from sentence_transformers import SentenceTransformer

model_args = RetrievalArgs(
    query_encoder = "hkunlp/instructor-base",
    pooling_method = ["mean", "dense"],
    dtype = "fp32"
)
retriever = DenseRetriever(**asdict(model_args), cache_dir=model_args.model_cache_dir)
tokenizer = retriever.tokenizer

with torch.no_grad():
    sent_model = SentenceTransformer(model_args.query_encoder, device="cpu")
    retriever.dense_pooler.weight.data = sent_model.state_dict()["2.linear.weight"]

    x = sent_model.encode(["I love you"])
    y = retriever.encode("I love you")
    print(torch.isclose(torch.from_numpy(x), y))
    retriever.save_pretrained("data/outputs/instructor-base")

Then we evaluate with

torchrun --nproc_per_node 8 -m evaluation.eval_popqa --query_encoder data/outputs/instructor-base/encoder --pooling_method mean dense --version instructor --data_root /data/llm-embedder

Leaderboard Copy Section

All the following results are based on meta-llama/Llama-27b-chat-hf with torch==2.0.1, transformers==4.30.0 on a 8xA100 machine with CUDA==11.4.

Research Llm Embedder Docs Fine Tune (`research/llm_embedder/docs/fine-tune.md`) Copy Section

Fine-tuning

Environment Copy Chapter

It is recommended that you create a new environment:

cd FlagEmbedding/llm_embedder

conda env create -f environment.yaml --name llm-embedder
conda activate llm-embedder

To use BM25, you must download java11 and anserini, then add java to your PATH:

# feel free to alternate /data to your prefered location
wget https://huggingface.co/datasets/namespace-Pt/projects/resolve/main/java11.tar.gz?download=true -O /data/java11.tar.gz
wget https://huggingface.co/datasets/namespace-Pt/projects/resolve/main/anserini.tar.gz?download=true -O /data/anserini.tar.gz

cd /data
tar -xzvf java11.tar.gz
tar -xzvf anserini.tar.gz

# below just temporarily set JAVA_HOME; it is RECOMMENDED that you store the lines the setting in ~/.bashrc
export JAVA_HOME=/data/jdk-11.0.2
export PATH=$JAVA_HOME/bin:$PATH

Data Copy Chapter

You should download the data for fine-tuning & evaluation then untar the file at anywhere you prefer, e.g. /data, which results in a folder /data/llm-embedder:

# feel free to alternate /data to your prefered location
wget https://huggingface.co/datasets/namespace-Pt/projects/resolve/main/llm-embedder.tar.gz?download=true -O /data/llm-embedder.tar.gz

cd /data
tar -xzvf llm-embedder-eval.tar.gz

The corpus of QReCC for conversational search is too large (54M passages), we separately upload it to huggingface datasets namespace-Pt/qrecc-corpus. To evaluate the performance on conversational search, you should load it and save it as json file in the qrecc folder:

import datasets
# load dataset
qrecc_corpus = datasets.load_dataset("namespace-Pt/qrecc-corpus", split="train")
# save to jsonline format in YOUR data folder
qrecc_corpus.to_json("/data/llm-embedder/convsearch/qrecc/corpus.json", force_ascii=False, lines=True, orient="records")

The data formats for training and evaluation are as follows:

# training
{
  "query": str,
  "pos": List[str],
  "neg": List[str],
  "pos_index": Optional[List[int]],         # Indices of the positives w.r.t. the corpus. When a global corpus is not available (e.g. long conversation), just ignore this field.
  "neg_index": Optional[List[int]],         # Indices of the negatives w.r.t. the corpus. When a global corpus is not available (e.g. long conversation), just ignore this field.
  "teacher_scores": Optional[List[float]],  # Scores from an LM or a reranker, used for distillation.
  "answers": Optional[List[str]],           # List of answers for the query, used for LM scoring.
}

# evaluation
{
  "query": str,
  "pos_index": Optional[List[int]],         # Indices of the positives w.r.t. corpus. When there is no positives pre-defined (e.g. NQ), just ignore this field.
  "answers": Optional[List[str]],           # List of answers for computing NQ metrics.
  "key": Optional[List[str]],               # Retrieval results of the query. Usually used for RAG or reranking.
  "key_index": Optional[List[int]],         # Key indices w.r.t. the corpus.
}

Retriever Copy Chapter

Below are several important arguments for training. The meaning and usage of other arguments can be inspected from code or running python run_dense.py --help from command line.

• train_data: required, one or a list of json files with the aforementioned formatting.
• eval_data: optional, one json file with the aforementioned formatting. If an eval_data is speficied, the trainer will automatically do evaluation on the eval_data.
• corpus: optional, the global corpus where positives.

IMPORTANT NOTE

• For any path specified for train_data, eval_data, and corpus: if it is prefixed with llm-embedder, it will be solved to the relative path against data_root. Note that you can modify the default value of data_root, so that you don't need to type it for each command.
• During fine-tuning, we save the output model in the huggingface transformers🤗 format. To use it from sentence_transformers, you should convert it to sentence_transformers checkpoint in advance:
  bash Copy

  python scripts/ours2st.py --encoder data/outputs/your-output-dir/encoder

  Then everything is the same as described in README.

LLM-Embedder (Multi-Task Fine-Tune) Copy Section

# Remember to modify the data_root to your data root in the script :)
bash scripts/llm-embedder.sh

Single Task Fine-Tune Copy Section

Below we provide commands to fine-tune a retriever on a single task.

QA

torchrun --nproc_per_node=8 run_dense.py \
--output_dir data/outputs/nq \
--train_data llm-embedder:qa/nq/train.json \
--eval_data llm-embedder:qa/nq/test.json \
--corpus llm-embedder:qa/nq/corpus.json \
--metrics nq \
--key_max_length 128 \
--query_max_length 32 \
--contrastive_weight 0 \
--stable_distill \
--eval_steps 2000 \
--save_steps 2000 \
--max_steps 2000 \
--data_root /data/llm-embedder

In-Context Learning

torchrun --nproc_per_node=8 run_dense.py \
--output_dir data/outputs/icl \
--train_data llm-embedder:icl/icl/train.json \
--select_positive random \
--contrastive_weight 0 \
--stable_distill \
--save_steps 6000 \
--max_steps 6000 \
--data_root /data/llm-embedder

Long-Range Language Modeling

torchrun --nproc_per_node=8 run_dense.py \
--output_dir data/outputs/lrlm \
--train_data llm-embedder:lrlm/books3/train.json llm-embedder:lrlm/arxiv/train.json llm-embedder:lrlm/codeparrot/train.json \
--select_positive teacher \
--teacher_scores_margin 0.1 \
--contrastive_weight 0 \
--teacher_temperature 0.1 \
--save_steps 4000 \
--max_steps 4000 \
--data_root /data/llm-embedder

Long Chat

torchrun --nproc_per_node=8 run_dense.py \
--output_dir data/outputs/msc \
--train_data llm-embedder:chat/msc/train.json \
--select_positive teacher \
--select_negative random \
--contrastive_weight 0 \
--teacher_temperature 0.1 \
--save_steps 4000 \
--max_steps 4000 \
--data_root /data/llm-embedder

Tool

torchrun --nproc_per_node=8 run_dense.py \
--output_dir data/outputs/tool \
--train_data llm-embedder:tool/toolbench/train.json \
--eval_data llm-embedder:tool/toolbench/test.json \
--corpus llm-embedder:tool/toolbench/corpus.json \
--key_template {text} \
--metrics ndcg \
--eval_steps 2000 \
--save_steps 2000 \
--max_steps 2000 \
--data_root /data/llm-embedder

Conversation Search

torchrun --nproc_per_node=8 run_dense.py \
--output_dir data/outputs/qrecc \
--train_data llm-embedder:conversation/qrecc/train.concat.json \
--eval_data llm-embedder:conversation/qrecc/test.concat.json \
--corpus llm-embedder:conversation/qrecc/corpus.json \
--key_template '{text}' \
--metrics mrr ndcg \
--cutoffs 3 10 100 \
--eval_steps 2000 \
--save_steps 2000 \
--max_steps 2000 \
--data_root /data/llm-embedder

Mine Negatives Copy Section

# BGE (the result will be saved at llm-embedder:qa/nq/train.neg.bge.json)
torchrun --nproc_per_node=8 -m evaluation.eval_retrieval \
--eval_data llm-embedder:qa/nq/train.json \
--corpus llm-embedder:qa/nq/corpus.json \
--metrics mrr recall collate_neg \
--save_name bge \
--data_root /data/llm-embedder

# BM25 (the result will be saved at llm-embedder:qa/nq/train.neg.bm25.json; anserini_dir is the folder where you untar anserini.tar.gz)
torchrun --nproc_per_node 8 -m evaluation.eval_retrieval \
--anserini_dir /data/anserini \
--retrieval_method bm25 \
--eval_data llm-embedder:qa/nq/train.json \
--corpus llm-embedder:qa/nq/corpus.json \
--metrics mrr recall collate_neg \
--save_name bm25 \
--data_root /data/llm-embedder

LM Scoring Copy Chapter

Score positives and negatives in eval_data with $p(o|q,k)$ where $o$ is the desired output (i.e. answers field), $q$ is the query, and $k$ is a key (could be positive or negative).

torchrun --nproc_per_node=8 run_lm_score.py \
--eval_data llm-embedder:qa/msmarco/train.json \
--data_root /data/llm-embedder \
--model_name_or_path meta-llama/Llama-2-7b-chat-hf \
--save_name llama2-7b-chat

Results will be saved at /data/llm-embedder/qa/msmarco/train.scored.llama2-7b-chat.json

Known Issues Copy Chapter

• transformers==4.30.0 raises error when using deepspeed schedulerconfig
  • modify line 1750 in trainer.py
  python Copy

  if use_accelerator_prepare:
        # NOTE: fix bug in transformers 4.30.0
        # model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
        self.model.train()
        if hasattr(self.lr_scheduler, "step"):
            if self.use_apex:
                model = self.accelerator.prepare(self.model)
            else:
                model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
        else:
            # to handle cases wherein we pass "DummyScheduler" such as when it is specified in DeepSpeed config.
            model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
                self.model, self.optimizer, self.lr_scheduler
            )

Source API Abc Evaluation (`docs/source/API/abc/evaluation.rst`) Copy Section

Evaluation

.. toctree::
evaluation/arguments
evaluation/data_loader
evaluation/searcher
evaluation/evaluator
evaluation/runner

Source API Abc Inference (`docs/source/API/abc/inference.rst`) Copy Section

Inference

.. toctree::
inference/AbsEmbedder
inference/AbsReranker

Source API Evaluation Airbench (`docs/source/API/evaluation/airbench.rst`) Copy Section

AIR-Bench

AIR-Bench <https://github.com/AIR-Bench/AIR-Bench>_ (Automated heterogeneous Information Retrieval Benchmark) is a dynamic (actively being updated) benchmark for information retrieval.
Now the benchmark contains two versions. Notice that the testing data is generated by LLMs with out human intervention.
This helps the evaluation of new domains easier and faster to be updated. It also makes it impossible for any models to have the test data covered in their training sets.

You can evaluate model's performance on AIR-Bench by running our provided shell script:

.. code:: bash

chmod +x /examples/evaluation/air_bench/eval_air_bench.sh
./examples/evaluation/air_bench/eval_air_bench.sh

Or by running:

.. code:: bash

python -m FlagEmbedding.evaluation.air_bench \
--benchmark_version AIR-Bench_24.05 \
--task_types qa long-doc \
--domains arxiv \
--languages en \
--splits dev test \
--output_dir ./air_bench/search_results \
--search_top_k 1000 \
--rerank_top_k 100 \
--cache_dir /root/.cache/huggingface/hub \
--overwrite False \
--embedder_name_or_path BAAI/bge-m3 \
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
--devices cuda:0 cuda:1 \
--model_cache_dir /root/.cache/huggingface/hub \
--reranker_max_length 1024

change the embedder, reranker, devices and cache directory to your preference.

.. toctree::
:hidden:

airbench/arguments
airbench/runner

Source API Evaluation Beir (`docs/source/API/evaluation/beir.rst`) Copy Section

BEIR

BEIR <https://github.com/beir-cellar/beir>_ (Benchmarking-IR) is a heterogeneous evaluation benchmark for information retrieval.
It is designed for evaluating the performance of NLP-based retrieval models and widely used by research of modern embedding models.

You can evaluate model's performance on the BEIR benchmark by running our provided shell script:

.. code:: bash

chmod +x /examples/evaluation/beir/eval_beir.sh
./examples/evaluation/beir/eval_beir.sh

Or by running:

.. code:: bash

python -m FlagEmbedding.evaluation.beir \
--eval_name beir \
--dataset_dir ./beir/data \
--dataset_names fiqa arguana cqadupstack \
--splits test dev \
--corpus_embd_save_dir ./beir/corpus_embd \
--output_dir ./beir/search_results \
--search_top_k 1000 \
--rerank_top_k 100 \
--cache_path /root/.cache/huggingface/hub \
--overwrite False \
--k_values 10 100 \
--eval_output_method markdown \
--eval_output_path ./beir/beir_eval_results.md \
--eval_metrics ndcg_at_10 recall_at_100 \
--ignore_identical_ids True \
--embedder_name_or_path BAAI/bge-large-en-v1.5 \
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
--devices cuda:0 cuda:1 \
--reranker_max_length 1024 \

change the embedder, devices and cache directory to your preference.

.. toctree::
:hidden:

beir/arguments
beir/data_loader
beir/evaluator
beir/runner

Source API Evaluation Miracl (`docs/source/API/evaluation/miracl.rst`) Copy Section

MIRACL

MIRACL <https://project-miracl.github.io/>_ (Multilingual Information Retrieval Across a Continuum of Languages)
is an WSDM 2023 Cup challenge that focuses on search across 18 different languages.
They release a multilingual retrieval dataset containing the train and dev set for 16 "known languages" and only dev set for 2 "surprise languages".
The topics are generated by native speakers of each language, who also label the relevance between the topics and a given document list.
You can found the dataset <https://huggingface.co/datasets/miracl/miracl-corpus>_ on HuggingFace.

You can evaluate model's performance on MIRACL simply by running our provided shell script:

.. code:: bash

chmod +x /examples/evaluation/miracl/eval_miracl.sh
./examples/evaluation/miracl/eval_miracl.sh

Or by running:

.. code:: bash

python -m FlagEmbedding.evaluation.miracl \
--eval_name miracl \
--dataset_dir ./miracl/data \
--dataset_names bn hi sw te th yo \
--splits dev \
--corpus_embd_save_dir ./miracl/corpus_embd \
--output_dir ./miracl/search_results \
--search_top_k 1000 \
--rerank_top_k 100 \
--cache_path /root/.cache/huggingface/hub \
--overwrite False \
--k_values 10 100 \
--eval_output_method markdown \
--eval_output_path ./miracl/miracl_eval_results.md \
--eval_metrics ndcg_at_10 recall_at_100 \
--embedder_name_or_path BAAI/bge-m3 \
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
--devices cuda:0 cuda:1 \
--cache_dir /root/.cache/huggingface/hub \
--reranker_max_length 1024

change the embedder, reranker, devices and cache directory to your preference.

.. toctree::
:hidden:

miracl/data_loader
miracl/runner

Source API Evaluation Mkqa (`docs/source/API/evaluation/mkqa.rst`) Copy Section

MKQA

MKQA <https://github.com/apple/ml-mkqa>_ is an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages.
The queries are sampled from the Google Natural Questions Dataset.

Each example in the dataset has the following structure:

.. code:: bash

{
    'example_id': 563260143484355911,
    'queries': {
        'en': "who sings i hear you knocking but you can't come in",
        'ru': "кто поет i hear you knocking but you can't come in",
        'ja': '「 I hear you knocking」は誰が歌っていますか',
        'zh_cn': "《i hear you knocking but you can't come in》是谁演唱的",
        ...
    },
    'query': "who sings i hear you knocking but you can't come in",
    'answers': {
        'en': [{
            'type': 'entity',
            'entity': 'Q545186',
            'text': 'Dave Edmunds',
            'aliases': [],
        }],
        'ru': [{
            'type': 'entity',
            'entity': 'Q545186',
            'text': 'Эдмундс, Дэйв',
            'aliases': ['Эдмундс', 'Дэйв Эдмундс', 'Эдмундс Дэйв', 'Dave Edmunds'],
        }],
        'ja': [{
            'type': 'entity',
            'entity': 'Q545186',
            'text': 'デイヴ・エドモンズ',
            'aliases': ['デーブ・エドモンズ', 'デイブ・エドモンズ'],
        }],
        'zh_cn': [{
            'type': 'entity', 
            'text': '戴维·埃德蒙兹 ', 
            'entity': 'Q545186',
        }],
        ...
    },
}

You can evaluate model's performance on MKQA simply by running our provided shell script:

.. code:: bash

chmod +x /examples/evaluation/mkqa/eval_mkqa.sh
./examples/evaluation/mkqa/eval_mkqa.sh

Or by running:

.. code:: bash

python -m FlagEmbedding.evaluation.mkqa \
--eval_name mkqa \
--dataset_dir ./mkqa/data \
--dataset_names en zh_cn \
--splits test \
--corpus_embd_save_dir ./mkqa/corpus_embd \
--output_dir ./mkqa/search_results \
--search_top_k 1000 \
--rerank_top_k 100 \
--cache_path /root/.cache/huggingface/hub \
--overwrite False \
--k_values 20 \
--eval_output_method markdown \
--eval_output_path ./mkqa/mkqa_eval_results.md \
--eval_metrics qa_recall_at_20 \
--embedder_name_or_path BAAI/bge-m3 \
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
--devices cuda:0 cuda:1 \
--cache_dir /root/.cache/huggingface/hub \
--reranker_max_length 1024

change the embedder, reranker, devices and cache directory to your preference.

.. toctree::
:hidden:

mkqa/data_loader
mkqa/evaluator
mkqa/runner

Source API Evaluation Mldr (`docs/source/API/evaluation/mldr.rst`) Copy Section

MLDR

MLDR <https://huggingface.co/datasets/Shitao/MLDR>_ is a Multilingual Long-Document Retrieval dataset built on Wikipeida, Wudao and mC4, covering 13 typologically diverse languages.
Specifically, we sample lengthy articles from Wikipedia, Wudao and mC4 datasets and randomly choose paragraphs from them.
Then we use GPT-3.5 to generate questions based on these paragraphs.
The generated question and the sampled article constitute a new text pair to the dataset.

An example of train set looks like:

.. code:: bash

{
    'query_id': 'q-zh-<...>', 
    'query': '...', 
    'positive_passages': [
        {
            'docid': 'doc-zh-<...>',
            'text': '...'
        }
    ],
    'negative_passages': [
        {
            'docid': 'doc-zh-<...>',
            'text': '...'
        },
        ...
    ]
}

An example of dev and test set looks like:

.. code:: bash

{
    'query_id': 'q-zh-<...>', 
    'query': '...', 
    'positive_passages': [
        {
            'docid': 'doc-zh-<...>',
            'text': '...'
        }
    ],
    'negative_passages': []
}

An example of corpus looks like:

.. code:: bash

{
    'docid': 'doc-zh-<...>', 
    'text': '...'
}

You can evaluate model's performance on MLDR simply by running our provided shell script:

.. code:: bash

chmod +x /examples/evaluation/mldr/eval_mldr.sh
./examples/evaluation/mldr/eval_mldr.sh

Or by running:

.. code:: bash

python -m FlagEmbedding.evaluation.mldr \
--eval_name mldr \
--dataset_dir ./mldr/data \
--dataset_names hi \
--splits test \
--corpus_embd_save_dir ./mldr/corpus_embd \
--output_dir ./mldr/search_results \
--search_top_k 1000 \
--rerank_top_k 100 \
--cache_path /root/.cache/huggingface/hub \
--overwrite False \
--k_values 10 100 \
--eval_output_method markdown \
--eval_output_path ./mldr/mldr_eval_results.md \
--eval_metrics ndcg_at_10 \
--embedder_name_or_path BAAI/bge-m3 \
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
--devices cuda:0 cuda:1 \
--cache_dir /root/.cache/huggingface/hub \
--embedder_passage_max_length 8192 \
--reranker_max_length 8192

change the args of embedder, reranker, devices and cache directory to your preference.

.. toctree::
:hidden:

mldr/data_loader
mldr/runner

Source API Evaluation Msmarco (`docs/source/API/evaluation/msmarco.rst`) Copy Section

MSMARCO

MS Marco <https://microsoft.github.io/msmarco/>_ (Microsoft MAchine Reading Comprehension) is a large scale real-world reading comprehension dataset.
It is widely used in information retrieval, question answering, and natural language processing research.

You can evaluate model's performance on MS MARCO simply by running our provided shell script:

.. code:: bash

chmod +x /examples/evaluation/msmarco/eval_msmarco.sh
./examples/evaluation/msmarco/eval_msmarco.sh

Or by running:

.. code:: bash

python -m FlagEmbedding.evaluation.msmarco \
--eval_name msmarco \
--dataset_dir ./msmarco/data \
--dataset_names passage \
--splits dev \
--corpus_embd_save_dir ./msmarco/corpus_embd \
--output_dir ./msmarco/search_results \
--search_top_k 1000 \
--rerank_top_k 100 \
--cache_path /root/.cache/huggingface/hub \
--overwrite True \
--k_values 10 100 \
--eval_output_method markdown \
--eval_output_path ./msmarco/msmarco_eval_results.md \
--eval_metrics ndcg_at_10 recall_at_100 \
--embedder_name_or_path BAAI/bge-large-en-v1.5 \
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
--devices cuda:0 cuda:1 cuda:2 cuda:3 cuda:4 cuda:5 cuda:6 cuda:7 \
--cache_dir /root/.cache/huggingface/hub \
--reranker_max_length 1024

change the embedder, reranker, devices and cache directory to your preference.

.. toctree::
:hidden:

msmarco/data_loader
msmarco/runner

Source API Evaluation Mteb (`docs/source/API/evaluation/mteb.rst`) Copy Section

MTEB

MTEB <https://github.com/embeddings-benchmark/mteb>_ (The Massive Text Embedding Benchmark) is a large-scale evaluation framework designed to assess the performance of text embedding models across a wide variety of NLP tasks.
Introduced to standardize and improve the evaluation of text embeddings, MTEB is crucial for assessing how well these models generalize across various real-world applications.
It contains a wide range of datasets in eight main NLP tasks and different languages, and provides an easy pipeline for evaluation.
It also holds the well known MTEB leaderboard <https://huggingface.co/spaces/mteb/leaderboard>_, which contains a ranking of the latest first-class embedding models.

You can evaluate model's performance on the whole MTEB benchmark by running our provided shell script:

.. code:: bash

chmod +x /examples/evaluation/mteb/eval_mteb.sh
./examples/evaluation/mteb/eval_mteb.sh

Or by running:

.. code:: bash

python -m FlagEmbedding.evaluation.mteb \
--eval_name mteb \
--output_dir ./mteb/search_results \
--languages eng \
--tasks NFCorpus BiorxivClusteringS2S SciDocsRR \
--eval_output_path ./mteb/mteb_eval_results.json \
--embedder_name_or_path BAAI/bge-large-en-v1.5 \
--devices cuda:7 \
--cache_dir /root/.cache/huggingface/hub

change the embedder, devices and cache directory to your preference.

.. toctree::
:hidden:

mteb/arguments
mteb/searcher
mteb/runner

Source API Finetune Embedder (`docs/source/API/finetune/embedder.rst`) Copy Section

Embedder

.. toctree::
embedder/encoder_only
embedder/decoder_only

Source API Finetune Reranker (`docs/source/API/finetune/reranker.rst`) Copy Section

Reranker

.. toctree::
reranker/encoder_only
reranker/decoder_only

Source API Inference FlagAutoModel (`docs/source/API/inference/FlagAutoModel.rst`) Copy Section

FlagAutoModel

.. autoclass:: FlagEmbedding.inference.FlagAutoModel

Methods Copy Chapter

.. automethod:: FlagEmbedding.inference.FlagAutoModel.from_finetuned

Source API Inference FlagAutoReranker (`docs/source/API/inference/FlagAutoReranker.rst`) Copy Section

FlagAutoReranker

.. autoclass:: FlagEmbedding.inference.FlagAutoReranker

Methods Copy Chapter

.. automethod:: FlagEmbedding.inference.FlagAutoReranker.from_finetuned

Source API Abc Evaluation Arguments (`docs/source/API/abc/evaluation/arguments.rst`) Copy Section

Arguments

.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvalArgs

.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvalModelArgs

Source API Abc Evaluation Data Loader (`docs/source/API/abc/evaluation/data_loader.rst`) Copy Section

dataset loader

.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.available_dataset_names
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.available_splits
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.check_dataset_names
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.check_splits
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.load_corpus
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.load_qrels
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.load_queries
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_remote_corpus
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_remote_qrels
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_remote_queries
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_local_corpus
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_local_qrels
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_local_queries
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._download_file
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._get_fpath_size
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._download_gz_file
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._download_zip_file

Source API Abc Evaluation Searcher (`docs/source/API/abc/evaluation/searcher.rst`) Copy Section

======== searcher

EvalRetriever

.. autoclass:: FlagEmbedding.abc.evaluation.EvalRetriever

EvalDenseRetriever

.. autoclass:: FlagEmbedding.abc.evaluation.EvalDenseRetriever

EvalReranker

.. autoclass:: FlagEmbedding.abc.evaluation.EvalReranker

Source API Abc Finetune Embedder (`docs/source/API/abc/finetune/embedder.rst`) Copy Section

Embedder

.. toctree::
embedder/AbsArguments
embedder/AbsDataset
embedder/AbsModeling
embedder/AbsTrainer
embedder/AbsRunner

Source API Abc Finetune Reranker (`docs/source/API/abc/finetune/reranker.rst`) Copy Section

Reranker

.. toctree::
reranker/AbsArguments
reranker/AbsDataset
reranker/AbsModeling
reranker/AbsTrainer
reranker/AbsRunner

Source API Abc Inference AbsEmbedder (`docs/source/API/abc/inference/AbsEmbedder.rst`) Copy Section

AbsEmbedder

.. autoclass:: FlagEmbedding.abc.inference.AbsEmbedder

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.get_target_devices

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.get_detailed_instruct

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_queries

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_corpus

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_single_device

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.start_multi_process_pool

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder._encode_multi_process_worker

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.stop_multi_process_pool

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_multi_process

.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder._concatenate_results_from_multi_process

Source API Abc Inference AbsReranker (`docs/source/API/abc/inference/AbsReranker.rst`) Copy Section

AbsReranker

.. autoclass:: FlagEmbedding.abc.inference.AbsReranker

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.get_target_devices

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.get_detailed_instruct

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.get_detailed_inputs

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.compute_score

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.compute_score_single_gpu

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.start_multi_process_pool

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.encode_multi_process

.. automethod:: FlagEmbedding.abc.inference.AbsReranker._encode_multi_process_worker

.. automethod:: FlagEmbedding.abc.inference.AbsReranker.stop_multi_process_pool

Source API Evaluation Airbench Arguments (`docs/source/API/evaluation/airbench/arguments.rst`) Copy Section

arguments

.. autoclass:: FlagEmbedding.evaluation.air_bench.AIRBenchEvalModelArgs

Source API Evaluation Airbench Runner (`docs/source/API/evaluation/airbench/runner.rst`) Copy Section

runner

.. autoclass:: FlagEmbedding.evaluation.air_bench.AIRBenchEvalRunner

Source API Evaluation Beir Arguments (`docs/source/API/evaluation/beir/arguments.rst`) Copy Section

arguments

.. autoclass:: FlagEmbedding.evaluation.beir.arguments.BEIREvalArgs

Source API Evaluation Beir Data Loader (`docs/source/API/evaluation/beir/data_loader.rst`) Copy Section

data loader

.. autoclass:: FlagEmbedding.evaluation.beir.data_loader.BEIREvalDataLoader

Source API Evaluation Beir Evaluator (`docs/source/API/evaluation/beir/evaluator.rst`) Copy Section

evaluator

.. autoclass:: FlagEmbedding.evaluation.beir.evaluator.BEIREvaluator

Source API Evaluation Miracl Data Loader (`docs/source/API/evaluation/miracl/data_loader.rst`) Copy Section

data_loader

.. autoclass:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader

Methods Copy Chapter

.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader.available_dataset_names
.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader.available_splits
.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader._load_remote_corpus
.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader._load_remote_qrels
.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader._load_remote_queries

Source API Evaluation Miracl Runner (`docs/source/API/evaluation/miracl/runner.rst`) Copy Section

runner

.. autoclass:: FlagEmbedding.evaluation.miracl.MIRACLEvalRunner
:members:

Source API Evaluation Mkqa Data Loader (`docs/source/API/evaluation/mkqa/data_loader.rst`) Copy Section

data_loader

.. autoclass:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader

Methods Copy Chapter

.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader.available_dataset_names
.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader.available_splits
.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader.load_corpus
.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_local_qrels
.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_remote_corpus
.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_remote_qrels
.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_remote_queries

Source API Evaluation Mkqa Evaluator (`docs/source/API/evaluation/mkqa/evaluator.rst`) Copy Section

evaluator

.. autoclass:: FlagEmbedding.evaluation.mkqa.MKQAEvaluator
:members:

Source API Evaluation Mkqa Runner (`docs/source/API/evaluation/mkqa/runner.rst`) Copy Section

runner

.. autoclass:: FlagEmbedding.evaluation.mkqa.MKQAEvalRunner
:members:

Source API Evaluation Mldr Data Loader (`docs/source/API/evaluation/mldr/data_loader.rst`) Copy Section

data_loader

.. autoclass:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader

Methods Copy Chapter

.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader.available_dataset_names
.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader.available_splits
.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader._load_remote_corpus
.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader._load_remote_qrels
.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader._load_remote_queries

Source API Evaluation Mldr Runner (`docs/source/API/evaluation/mldr/runner.rst`) Copy Section

runner

.. autoclass:: FlagEmbedding.evaluation.mldr.MLDREvalRunner
:members:

Source API Evaluation Msmarco Data Loader (`docs/source/API/evaluation/msmarco/data_loader.rst`) Copy Section

data_loader

.. autoclass:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader

Methods Copy Chapter

.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader.available_dataset_names
.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader.available_splits
.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader._load_remote_corpus
.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader._load_remote_qrels
.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader._load_remote_queries

Source API Evaluation Msmarco Runner (`docs/source/API/evaluation/msmarco/runner.rst`) Copy Section

runner

.. autoclass:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalRunner
:members:

Source API Evaluation Mteb Arguments (`docs/source/API/evaluation/mteb/arguments.rst`) Copy Section

arguments

.. autoclass:: FlagEmbedding.evaluation.mteb.arguments.MTEBEvalArgs

Source API Evaluation Mteb Runner (`docs/source/API/evaluation/mteb/runner.rst`) Copy Section

runner

.. autoclass:: FlagEmbedding.evaluation.mteb.runner.MTEBEvalRunner

Source API Evaluation Mteb Searcher (`docs/source/API/evaluation/mteb/searcher.rst`) Copy Section

searcher

.. autoclass:: FlagEmbedding.evaluation.mteb.searcher.MTEBEvalDenseRetriever

.. autoclass:: FlagEmbedding.evaluation.mteb.searcher.MTEBEvalReranker

Source API Finetune Embedder Decoder Only (`docs/source/API/finetune/embedder/decoder_only.rst`) Copy Section

Decoder Only

.. toctree::
decoder_only/base
decoder_only/icl

Source API Finetune Embedder Encoder Only (`docs/source/API/finetune/embedder/encoder_only.rst`) Copy Section

Encoder Only

.. toctree::
encoder_only/base
encoder_only/m3

Source API Finetune Reranker Decoder Only (`docs/source/API/finetune/reranker/decoder_only.rst`) Copy Section

Decoder Only

.. toctree::
decoder_only/base
decoder_only/layerwise

Source API Inference Embedder Embedder (`docs/source/API/inference/embedder/embedder.rst`) Copy Section

Embedder

.. toctree::
encoder_only/BaseEmbedder
encoder_only/M3Embedder
decoder_only/BaseLLMEmbedder
decoder_only/ICLLLMEmbedder

Source API Inference Reranker Reranker (`docs/source/API/inference/reranker/reranker.rst`) Copy Section

Reranker

.. toctree::
encoder_only/BaseReranker
decoder_only/BaseLLMReranker
decoder_only/LayerWiseLLMReranker
decoder_only/LightweightLLMReranker

Source API Abc Finetune Embedder AbsArguments (`docs/source/API/abc/finetune/embedder/AbsArguments.rst`) Copy Section

AbsArguments

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModelArguments

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerDataArguments

Source API Abc Finetune Embedder AbsDataset (`docs/source/API/abc/finetune/embedder/AbsDataset.rst`) Copy Section

========== AbsDataset

AbsEmbedderTrainDataset

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainDataset

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainDataset._load_dataset

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainDataset._shuffle_text

AbsEmbedderCollator

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderCollator

AbsEmbedderSameDatasetTrainDataset

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset.refresh_epoch

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._load_dataset

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._get_file_batch_size

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._get_train_group_size

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._create_batch_data

AbsEmbedderSameDatasetCollator

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetCollator

EmbedderTrainerCallbackForDataRefresh

.. autoclass:: FlagEmbedding.abc.finetune.embedder.EmbedderTrainerCallbackForDataRefresh

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.embedder.EmbedderTrainerCallbackForDataRefresh.on_epoch_end

Source API Abc Finetune Embedder AbsModeling (`docs/source/API/abc/finetune/embedder/AbsModeling.rst`) Copy Section

=========== AbsModeling

AbsEmbedderModel

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.encode

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.compute_loss

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.compute_score

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.save

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.get_local_score

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.compute_local_score

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.forward

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.distill_loss

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._compute_no_in_batch_neg_loss

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._compute_in_batch_neg_loss

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._compute_cross_device_neg_loss

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._dist_gather_tensor

EmbedderOutput

.. autoclass:: FlagEmbedding.abc.finetune.embedder.EmbedderOutput

Source API Abc Finetune Embedder AbsRunner (`docs/source/API/abc/finetune/embedder/AbsRunner.rst`) Copy Section

========= AbsRunner

AbsEmbedderTrainer

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_tokenizer_and_model

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_trainer

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_train_dataset

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_data_collator

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.run

Source API Abc Finetune Embedder AbsTrainer (`docs/source/API/abc/finetune/embedder/AbsTrainer.rst`) Copy Section

========== AbsTrainer

AbsEmbedderTrainer

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainer

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainer.compute_loss

Source API Abc Finetune Reranker AbsArguments (`docs/source/API/abc/finetune/reranker/AbsArguments.rst`) Copy Section

AbsArguments

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModelArguments

.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderDataArguments

Source API Abc Finetune Reranker AbsDataset (`docs/source/API/abc/finetune/reranker/AbsDataset.rst`) Copy Section

========== AbsDataset

AbsRerankerTrainDataset

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset.create_one_example

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset._load_dataset

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset._shuffle_text

AbsRerankerCollator

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerCollator

AbsLLMRerankerTrainDataset

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsLLMRerankerTrainDataset

AbsLLMRerankerCollator

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsLLMRerankerCollator

Source API Abc Finetune Reranker AbsModeling (`docs/source/API/abc/finetune/reranker/AbsModeling.rst`) Copy Section

=========== AbsModeling

AbsRerankerModel

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.encode

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.gradient_checkpointing_enable

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.enable_input_require_grads

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.forward

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.compute_loss

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.save

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.save_pretrained

RerankerOutput

.. autoclass:: FlagEmbedding.abc.finetune.reranker.RerankerOutput

Source API Abc Finetune Reranker AbsRunner (`docs/source/API/abc/finetune/reranker/AbsRunner.rst`) Copy Section

========= AbsRunner

AbsRerankerTrainer

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_tokenizer_and_model

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_trainer

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_train_dataset

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_data_collator

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.run

Source API Abc Finetune Reranker AbsTrainer (`docs/source/API/abc/finetune/reranker/AbsTrainer.rst`) Copy Section

========== AbsTrainer

AbsRerankerTrainer

.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainer

Methods Copy Chapter

.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainer.compute_loss

Source API Finetune Embedder Decoder Only Base (`docs/source/API/finetune/embedder/decoder_only/base.rst`) Copy Section

Base

.. toctree::
base/arguments
base/modeling
base/runner
base/trainer

Source API Finetune Embedder Decoder Only Icl (`docs/source/API/finetune/embedder/decoder_only/icl.rst`) Copy Section

ICL

.. toctree::
icl/arguments
icl/dataset
icl/modeling
icl/runner
icl/trainer

Source API Finetune Embedder Encoder Only M3 (`docs/source/API/finetune/embedder/encoder_only/m3.rst`) Copy Section

M3

.. toctree::
m3/arguments
m3/modeling
m3/runner
m3/trainer

Source API Finetune Reranker Decoder Only Base (`docs/source/API/finetune/reranker/decoder_only/base.rst`) Copy Section

Base

.. toctree::
base/arguments
base/modeling
base/runner
base/trainer

Source API Finetune Reranker Decoder Only Layerwise (`docs/source/API/finetune/reranker/decoder_only/layerwise.rst`) Copy Section

Layerwise

.. toctree::
layerwise/arguments
layerwise/modeling
layerwise/runner
layerwise/trainer

--- METRICS ---

• Files Extracted: 93
• Estimated Token Budget: ~33381 tokens
• Canonical Reference: https://codewiki.google/github.com/FlagOpen/FlagEmbedding