# Technical Documentation: facebookresearch/dinov2 > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/facebookresearch/dinov2) ### Overview (`README.md`) :new: [2025-12-18] *Added support for loading XRay-DINO backbone following [Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning](https://arxiv.org/pdf/2405.01469), more details are [here](#pretrained-backbone-xray-dino)* :new: [2025-12-16] *Added Channel-Adaptive DINO code following [Scaling Channel-Adaptive Self-Supervised Learning](https://openreview.net/forum?id=pT8sgtRVAf), more details are [here](#dinov2-for-biology)* :new: [2025-12-16] *Added Cell-DINO code following [Cell-DINO: Self-Supervised Image-based Embeddings for Cell Fluorescent Microscopy](to appear in Plos One Computational Biology), more details are [here](#dinov2-for-biology)* [2025-08-14] *Please check out the more recent [DINOv3](https://github.com/facebookresearch/dinov3) effort continuing this line of work.* [2025-06-11] *Added dino.txt inference code, following [DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment](https://arxiv.org/abs/2412.16334).* [2023-10-26] *Added DINOv2 backbones with registers, following [Vision Transformers Need Registers](https://arxiv.org/abs/2309.16588).* # DINOv2: Learning Robust Visual Features without Supervision **[Meta AI Research, FAIR](https://ai.facebook.com/research/)** Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Patrick Labatut, Armand Joulin, Piotr Bojanowski [[`Paper #1`](https://arxiv.org/abs/2304.07193)] [`Paper #2`](https://arxiv.org/abs/2309.16588)] [[`Blog`](https://ai.facebook.com/blog/dino-v2-computer-vision-self-supervised-learning/)] [[`Demo`](https://dinov2.metademolab.com)] [[`BibTeX`](#citing-dinov2)] PyTorch implementation and pretrained models for DINOv2. For details, see the papers: **[DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/abs/2304.07193)** and **[Vision Transformers Need Registers](https://arxiv.org/abs/2309.16588)**. DINOv2 models produce high-performance visual features that can be directly employed with classifiers as simple as linear layers on a variety of computer vision tasks; these visual features are robust and perform well across domains without any requirement for fine-tuning. The models were pretrained on a dataset of 142 M images without using any labels or annotations. https://github.com/facebookresearch/dinov2/assets/60359573/f168823e-7922-415a-b429-578badf5c356
Visualization of the three first principal components of the patch features of all frames, mapped to RGB values.
## Pretrained models
model # of
params		 with
registers		 ImageNet
k-NN		 ImageNet
linear		 download
ViT-S/14 distilled 21 M :x: 79.0% 81.1% backbone only
ViT-S/14 distilled 21 M :white_check_mark: 79.1% 80.9% backbone only
ViT-B/14 distilled 86 M :x: 82.1% 84.5% backbone only
ViT-B/14 distilled 86 M :white_check_mark: 82.0% 84.6% backbone only
ViT-L/14 distilled 300 M :x: 83.5% 86.3% backbone only
ViT-L/14 distilled 300 M :white_check_mark: 83.8% 86.7% backbone only
ViT-g/14 1,100 M :x: 83.5% 86.5% backbone only
ViT-g/14 1,100 M :white_check_mark: 83.7% 87.1% backbone only
### Pretrained backbones (via PyTorch Hub) Please follow the instructions [here](https://pytorch.org/get-started/locally/) to install PyTorch (the only required dependency for loading the model). Installing PyTorch with CUDA support is strongly recommended. A corresponding [model card](MODEL_CARD.md) is included in the repository. ```python import torch # DINOv2 dinov2_vits14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14') dinov2_vitb14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14') dinov2_vitl14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14') dinov2_vitg14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14') # DINOv2 with registers dinov2_vits14_reg = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14_reg') dinov2_vitb14_reg = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14_reg') dinov2_vitl14_reg = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14_reg') dinov2_vitg14_reg = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14_reg') ``` ### Pretrained backbone: XRay-DINO Request for downloading the model is here: https://ai.meta.com/resources/models-and-libraries/raydino-downloads/ After filling the form, you will get an email with a temporary link. You can either download it using `wget` and indicate the checkpoint path in your local filesystem, or you can directly use the URL from the email in the following code: ```python import torch REPO_DIR = xray_dino_vitl16 = torch.hub.load(REPO_DIR, 'xray_dino_vitl16', source='local', weights=) ``` **License** Model weights are released under the FAIR Noncommercial Research License. See LICENSE_XRAY_DINO_MODEL for additional details. ### Pretrained heads - Image classification
backbone with
registers		 download
ImageNet
ViT-S/14 distilled :x: linear head (1 layer, 4 layers)
ViT-S/14 distilled :white_check_mark: linear head (1 layer, 4 layers)
ViT-B/14 distilled :x: linear head (1 layer, 4 layers)
ViT-B/14 distilled :white_check_mark: linear head (1 layer, 4 layers)
ViT-L/14 distilled :x: linear head (1 layer, 4 layers)
ViT-L/14 distilled :white_check_mark: linear head (1 layer, 4 layers)
ViT-g/14 :x: linear head (1 layer, 4 layers)
ViT-g/14 :white_check_mark: linear head (1 layer, 4 layers)
The (full) classifier models can be loaded via PyTorch Hub: ```python import torch # DINOv2 dinov2_vits14_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14_lc') dinov2_vitb14_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14_lc') dinov2_vitl14_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14_lc') dinov2_vitg14_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14_lc') # DINOv2 with registers dinov2_vits14_reg_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14_reg_lc') dinov2_vitb14_reg_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14_reg_lc') dinov2_vitl14_reg_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14_reg_lc') dinov2_vitg14_reg_lc = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14_reg_lc') ``` ### Pretrained heads - Depth estimation
backbone download head
NYUd KITTI
ViT-S/14 distilled linear (1 layer, 4 layers), DPT linear (1 layer, 4 layers), DPT
ViT-B/14 distilled linear (1 layer, 4 layers), DPT linear (1 layer, 4 layers), DPT
ViT-L/14 distilled linear (1 layer, 4 layers), DPT linear (1 layer, 4 layers), DPT
ViT-g/14 linear (1 layer, 4 layers), DPT linear (1 layer, 4 layers), DPT
### Pretrained heads - Semantic segmentation
backbone download model download head
ADE20K ADE20K VOC2012
ViT-S/14 distilled linear, multi-scale linear, multi-scale
ViT-B/14 distilled linear, multi-scale linear, multi-scale
ViT-L/14 distilled linear, multi-scale linear, multi-scale
ViT-g/14 Mask2Former linear, multi-scale linear, multi-scale
### Pretrained heads - Zero-shot tasks with dino.txt
backbone with
registers		 download
ViT-L/14 distilled :white_check_mark: vision head, text model, vocabulary, vocabulary license
The (full) dino.txt model can be loaded via PyTorch Hub: ```python import torch # DINOv2 dinov2_vitl14_reg4_dinotxt_tet1280d20h24l = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14_reg4_dinotxt_tet1280d20h24l') ``` ## Installation The training and evaluation code requires PyTorch 2.0 and [xFormers](https://github.com/facebookresearch/xformers) 0.0.18 as well as a number of other 3rd party packages. Note that the code has only been tested with the specified versions and also expects a Linux environment. To setup all the required dependencies for training and evaluation, please follow the instructions below: *[conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html)* **(Recommended)** - Clone the repository and then create and activate a `dinov2` conda environment using the provided environment definition: ```shell conda env create -f conda.yaml conda activate dinov2 ``` *[pip](https://pip.pypa.io/en/stable/getting-started/)* - Clone the repository and then use the provided `requirements.txt` to install the dependencies: ```shell pip install -r requirements.txt ``` For dense tasks (depth estimation and semantic segmentation), there are additional dependencies (specific versions of `mmcv` and `mmsegmentation`) which are captured in the `extras` dependency specifications: *[conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html)* **(Recommended)**: ```shell conda env create -f conda-extras.yaml conda activate dinov2-extras ``` *[pip](https://pip.pypa.io/en/stable/getting-started/)*: ```shell pip install -r requirements.txt -r requirements-extras.txt ``` ## Data preparation ### ImageNet-1k The root directory of the dataset should hold the following contents: - `/test/ILSVRC2012_test_00000001.JPEG` - `/test/[..]` - `/test/ILSVRC2012_test_00100000.JPEG` - `/train/n01440764/n01440764_10026.JPEG` - `/train/[...]` - `/train/n15075141/n15075141_9993.JPEG` - `/val/n01440764/ILSVRC2012_val_00000293.JPEG` - `/val/[...]` - `/val/n15075141/ILSVRC2012_val_00049174.JPEG` - `/labels.txt` The provided dataset implementation expects a few additional metadata files to be present under the extra directory: - `/class-ids-TRAIN.npy` - `/class-ids-VAL.npy` - `/class-names-TRAIN.npy` - `/class-names-VAL.npy` - `/entries-TEST.npy` - `/entries-TRAIN.npy` - `/entries-VAL.npy` These metadata files can be generated (once) with the following lines of Python code: ```python from dinov2.data.datasets import ImageNet for split in ImageNet.Split: dataset = ImageNet(split=split, root="", extra="") dataset.dump_extra() ``` Note that the root and extra directories do not have to be distinct directories. ### ImageNet-22k Please adapt the [dataset class](dinov2/data/datasets/image_net_22k.py) to match your local setup.
:warning: To execute the commands provided in the next sections for training and evaluation, the `dinov2` package should be included in the Python module search path, i.e. simply prefix the command to run with `PYTHONPATH=.`. ## Training ### Fast setup: training DINOv2 ViT-L/16 on ImageNet-1k Run DINOv2 training on 4 A100-80GB nodes (32 GPUs) in a SLURM cluster environment with submitit: ```shell python dinov2/run/train/train.py \ --nodes 4 \ --config-file dinov2/configs/train/vitl16_short.yaml \ --output-dir \ train.dataset_path=ImageNet:split=TRAIN:root=:extra= ``` Training time is approximately 1 day and the resulting checkpoint should reach 81.6% on k-NN eval and 82.9% on linear eval. The training code saves the weights of the teacher in the `eval` folder every 12500 iterations for evaluation. ### Long setup: training DINOv2 ViT-L/14 on ImageNet-22k Run DINOv2 training on 12 A100-80GB nodes (96 GPUs) in a SLURM cluster environment with submitit: ```shell python dinov2/run/train/train.py \ --nodes 12 \ --config-file dinov2/configs/train/vitl14.yaml \ --output-dir \ train.dataset_path=ImageNet22k:root=:extra= ``` Training time is approximately 3.3 days and the resulting checkpoint should reach 82.0% on k-NN eval and 84.5% on linear eval. The training code saves the weights of the teacher in the `eval` folder every 12500 iterations for evaluation. ## Evaluation The training code regularly saves the teacher weights. In order to evaluate the model, run the following evaluation on a single node: ### k-NN classification on ImageNet-1k ```shell python dinov2/run/eval/knn.py \ --config-file /config.yaml \ --pretrained-weights /eval/training_24999/teacher_checkpoint.pth \ --output-dir /eval/training_24999/knn \ --train-dataset ImageNet:split=TRAIN:root=:extra= \ --val-dataset ImageNet:split=VAL:root=:extra= ``` ### Logistic regression classification on ImageNet-1k ```shell python dinov2/run/eval/log_regression.py \ --config-file /config.yaml \ --pretrained-weights /eval/training_24999/teacher_checkpoint.pth \ --output-dir /eval/training_24999/logreg \ --train-dataset ImageNet:split=TRAIN:root=:extra= \ --val-dataset ImageNet:split=VAL:root=:extra= ``` ### Linear classification with data augmentation on ImageNet-1k ```shell python dinov2/run/eval/linear.py \ --config-file /config.yaml \ --pretrained-weights /eval/training_24999/teacher_checkpoint.pth \ --output-dir /eval/training_24999/linear \ --train-dataset ImageNet:split=TRAIN:root=:extra= \ --val-dataset ImageNet:split=VAL:root=:extra= ``` We release the weights from evaluating the different models:
model with
registers		 ImageNet
top-1		 linear evaluation
ViT-S/14 distilled :x: 81.1% linear head weights
ViT-S/14 distilled :white_check_mark: 80.8% linear head weights
ViT-B/14 distilled :x: 84.5% linear head weights
ViT-B/14 distilled :white_check_mark: 84.4% linear head weights
ViT-L/14 distilled :x: 86.3% linear head weights
ViT-L/14 distilled :white_check_mark: 86.5% linear head weights
ViT-g/14 :x: 86.5% linear head weights
ViT-g/14 :white_check_mark: 87.0% linear head weights
The performance of the provided pretrained model weights can be evaluated as follows on ImageNet-1k: ```shell python dinov2/run/eval/linear.py \ --config-file dinov2/configs/eval/vitg14_pretrain.yaml \ --pretrained-weights https://dl.fbaipublicfiles.com/dinov2/dinov2_vitg14/dinov2_vitg14_pretrain.pth \ --train-dataset ImageNet:split=TRAIN:root=:extra= \ --val-dataset ImageNet:split=VAL:root=:extra= ``` ## Notebooks A few notebooks are provided to help the community leverage the models and code:
  • Depth estimation - How to load and use the depth heads in combination with a matching backbone via mmcv
  • Semantic segmentation - How to load and use the segmentation heads in combination with a matching backbone via mmcv, and also how to load and use the Mask2Former-based segmentation model trained on ADE20K
## License DINOv2 code and model weights are released under the Apache License 2.0. See [LICENSE](LICENSE) for additional details. ## Contributing See [contributing](CONTRIBUTING.md) and the [code of conduct](CODE_OF_CONDUCT.md). ## Citing DINOv2 If you find this repository useful, please consider giving a star :star: and citation :t-rex:: ``` @misc{oquab2023dinov2, title={DINOv2: Learning Robust Visual Features without Supervision}, author={Oquab, Maxime and Darcet, Timothée and Moutakanni, Theo and Vo, Huy V. and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaaeldin and Howes, Russell and Huang, Po-Yao and Xu, Hu and Sharma, Vasu and Li, Shang-Wen and Galuba, Wojciech and Rabbat, Mike and Assran, Mido and Ballas, Nicolas and Synnaeve, Gabriel and Misra, Ishan and Jegou, Herve and Mairal, Julien and Labatut, Patrick and Joulin, Armand and Bojanowski, Piotr}, journal={arXiv:2304.07193}, year={2023} } ``` ``` @misc{darcet2023vitneedreg, title={Vision Transformers Need Registers}, author={Darcet, Timothée and Oquab, Maxime and Mairal, Julien and Bojanowski, Piotr}, journal={arXiv:2309.16588}, year={2023} } ``` ``` @misc{jose2024dinov2meetstextunified, title={DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment}, author={Cijo Jose and Théo Moutakanni and Dahyun Kang and Federico Baldassarre and Timothée Darcet and Hu Xu and Daniel Li and Marc Szafraniec and Michaël Ramamonjisoa and Maxime Oquab and Oriane Siméoni and Huy V. Vo and Patrick Labatut and Piotr Bojanowski}, journal={arXiv:2412.16334}, year={2024} } ``` # DINOv2 for Biology The contents of the source code contained in the cell_dino folders, including the code and model weights, are intended for research use only. It is not for use in medical procedures, including any diagnostics, treatment, or curative applications. Do not use this model for any clinical purpose or as a substitute for professional medical judgement. ## Scaling Channel-Adaptive Self-Supervised Learning (Channel-Adaptive DINO) [[`Paper `](https://openreview.net/forum?id=pT8sgtRVAf))] [[`BibTeX`](#citing-channeladaptivedino-and-dinov2)] Alice V. De Lorenci, Seungeun Yi, Théo Moutakanni, Piotr Bojanowski, Camille Couprie, Juan C. Caicedo, Wolfgang M. Pernice, with special thanks to Elouan Gardes for his contributions to the codebase. [README](https://github.com/facebookresearch/dinov2/blob/main/docs/README_CHANNEL_ADAPTIVE_DINO.md) ## Cell-DINO: Self-Supervised Image-based Embeddings for Cell Fluorescent Microscopy (Cell-DINO) Théo Moutakanni, Camille Couprie, Seungeun Yi, Elouan Gardes, Piotr Bojanowski, Hugo Touvron, Michael Doron, Zitong S. Chen, Nikita Moshkov, Mathilde Caron, Armand Joulin, Wolfgang M. Pernice, Juan C. Caicedo to appear soon. [README](https://github.com/facebookresearch/dinov2/blob/main/docs/README_CELL_DINO.md) ## Pretrained models ℹ️ Please follow the link provided below to get access to all the model weights: once accepted, an e-mail will be sent with the complete list of URLs pointing to all the available model weights. These URLs can then be used to either: - download the model or adapter weights to a local filesystem and point `torch.hub.load()` to these local weights via the `pretrained_path` parameters, or - directly invoke `torch.hub.load()` to download and load a backbone from its URL via also the `pretrained_url` parameter. ⚠️ Please use wget instead of a web browser to download the weights. **Download link:** https://ai.meta.com/resources/models-and-libraries/cell-dino-downloads/ ```python import torch REPO_DIR = # You can either download the URL link first, then load: cell_dino_vits8 = torch.hub.load(REPO_DIR, 'cell_dino_cp_vits8', source='local', pretrained_path=) # Or directly download the URL while using `torch.hub.load`: cell_dino_vits8 = torch.hub.load(REPO_DIR, 'cell_dino_cp_vits8', source='local', pretrained_url=) # Similarily for other models: cell_dino_vitl16_hpa_sc = torch.hub.load(REPO_DIR, 'cell_dino_hpa_vitl16', source='local', pretrained_path=) cell_dino_vitl16_hpa_fov = torch.hub.load(REPO_DIR, 'cell_dino_hpa_vitl16', source='local', pretrained_path=) channel_adaptive_dino_vitl16 = torch.hub.load(REPO_DIR, 'channel_adaptive_dino_vitl16', source='local', pretrained_path=) cell_dino_vitl14 = torch.hub.load(REPO_DIR, 'cell_dino_hpa_vitl14', source='local', pretrained_path=) ``` ## Licenses Code is released under the CC BY NC License. See [LICENSE_CELL_DINO_CODE](LICENSE_CELL_DINO_CODE) for additional details. Model weights are released under the FAIR Noncommercial Research License. See [LICENSE_CELL_DINO_CODE_WEIGHTS](LICENSE_CELL_DINO_CODE_WEIGHTS) for additional details. --- ### README CELL DINO (`docs/README_CELL_DINO.md`) # Cell-DINO: Self-Supervised Image-based Embeddings for Cell Fluorescent Microscopy Théo Moutakanni*, Camille Couprie*, Seungeun Yi*, Elouan Gardes*, Piotr Bojanowski*, Hugo Touvron*, Michael Doron, Zitong S. Chen, Nikita Moshkov, Mathilde Caron, Armand Joulin, Wolfgang M. Pernice, Juan C. Caicedo [[`BibTeX`](#citing-cell-dino)] **[*Meta AI Research, FAIR](https://ai.facebook.com/research/)** PyTorch implementation and pretrained models for Cell-DINO. The contents of this repo, including the code and model weights, are intended for research use only. It is not for use in medical procedures, including any diagnostics, treatment, or curative applications. Do not use this model for any clinical purpose or as a substitute for professional medical judgement. ## Pretrained models One model pretrained on HPA single cell, one model pretrained on HPA Field of View, one model pretrained on HPA Field of View at a larger resolution and one model pretrained on the combined cell painting dataset are released, see dowload instructions at the end of README.md, section "DINO for Biology". ## Installation Follow instructions in the DINOv2 README or build the following environment: ```shell conda create -n py310 python=3.10 conda activate py310 pip install -r requirements.txt pip install -U scikit-learn ``` ## Data preparation The HPA-FoV and HPA single cell (HPAone) datasets are available [here](https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD2443) The folder also contains three sample images ("sample_images") that can be dowloaded to test quickly inference in the provided notebook in notebooks/cell_dino THe required files for HPA single cell (dataloader : HPAone.py) are: fixed_size_masked_single_cells_HPA varied_size_masked_single_cells_HPA varied_size_masked_single_cells_pretrain_20240507.csv fixed_size_masked_single_cells_evaluation_20240507.csv fixed_size_masked_single_cells_pretrain_20240507.csv :warning: To execute the commands provided in the next sections for training and evaluation, the `dinov2` package should be included in the Python module search path, i.e. simply prefix the command to run with `PYTHONPATH=.`. ## Training ### Fast setup: training DINOv2 ViT-L/16 on HPA single cell dataset Run CellDINO training on 4 A100-80GB nodes (32 GPUs) in a SLURM cluster environment with submitit: ```shell python dinov2/run/train/train.py \ --nodes 4 \ --config-file dinov2/configs/train/cell_dino/vitl16_hpaone.yaml \ --output-dir \ train.dataset_path=HPAone:split=ALL:root= ``` Training time is approximately 2 days on 4 A100 GPU nodes and the resulting checkpoint should reach 78.5 F1 accuracy for protein localization with a linear evaluation. The training code saves the weights of the teacher in the `eval` folder every 9000 iterations for evaluation. ## Evaluation The training code regularly saves the teacher weights. In order to evaluate the model, run the following evaluation on a single node: ### Linear classification with data augmentation on HPAone or HPAFov (replace 'HPAone' by 'HPAFoV'): ```shell PYTHONPATH=.:dinov2/data python dinov2/run/eval/cell_dino/linear.py \ --config-file dinov2/configs/eval/cell_dino/vitl16_pretrain.yaml \ --pretrained-weights /eval/training_44999/teacher_checkpoint.pth \ --output-dir /eval/training_44999/linear \ --train-dataset HPAone:split=TRAIN:mode=PROTEIN_LOCALIZATION:root= \ --val-dataset HPAone:split=VAL:mode=PROTEIN_LOCALIZATION:root \ --val-metric-type mean_per_class_multilabel_f1 \ --loss-type binary_cross_entropy \ --avgpool \ ``` We release the weights from evaluating the different models: The performance of the provided pretrained model weights can be evaluated as follows on HPAone and HPAFoV (replace 'HPAone' by 'HPAFoV', and make sure to load the correct model weights) for the protein localization task: ```shell PYTHONPATH=.:dinov2/data python dinov2/run/eval/linear_celldino.py \ --config-file dinov2/configs/eval/cell_dino/vitl16_pretrain.yaml \ --pretrained-weights \ --output-dir \ --train-dataset HPAone:split=TRAIN:mode=PROTEIN_LOCALIZATION:root= \ --val-dataset HPAone:split=VAL:mode=PROTEIN_LOCALIZATION:root=root \ --val-metric-type mean_per_class_multilabel_f1 \ --loss-type binary_cross_entropy \ --avgpool \ ``` and ```shell PYTHONPATH=.:dinov2/data python dinov2/run/eval/cell_dino/linear.py \ --config-file dinov2/configs/eval/cell_dino/vitl16_pretrain.yaml \ --pretrained-weights \ --output-dir \ --train-dataset HPAone:split=TRAIN:mode=CELL_TYPE:root= \ --val-dataset HPAone:split=VAL:mode=CELL_TYPE:root=root \ --val-metric-type mean_per_class_multiclass_f1 \ --avgpool \ ``` for the cell line classification task. ### knn evaluation on HPAone: ```shell PYTHONPATH=.:dinov2/data python dinov2/run/eval/cell_dino/knn.py \ --config-file dinov2/configs/eval/cell_dino/vitl16_pretrain.yaml \ --pretrained-weights \ --output-dir \ --train-dataset HPAone:split=TRAIN:mode=CELL_TYPE:root= \ --val-dataset HPAone:split=VAL:mode=CELL_TYPE:root= \ --metric-type mean_per_class_multiclass_f1 \ --crop-size 384 \ --batch-size 256 \ --resize-size 0 \ --nb_knn 1 \ ``` for the cell line classification task. For the knn evaluation on HPAFoV, replace 'HPAone' by 'HPAFoV' in the command above. For protein localization, don't forget to change the metric type to mean_per_class_multilabel_f1. ## License Cell-DINO code is released under the CC by NC licence See [LICENSE_CELL_DINO_CODE](LICENSE_CELL_DINO_CODE) for additional details. Model weights will be released under the FAIR Non-Commercial Research License. See [LICENSE_CELL_DINO_MODELS](LICENSE_CELL_DINO_MODELS) for additional details. ## Contributing See [contributing](CONTRIBUTING.md) and the [code of conduct](CODE_OF_CONDUCT.md). ## Citing Cell-DINO If you find this repository useful, please consider giving a star :star: and citation :t-rex:: ``` @misc{, title={Cell-DINO: Self-Supervised Image-based Embeddings for Cell Fluorescent Microscopy}, author={Moutakanni, Th\'eo and Couprie, Camille and Yi, Seungeun and Gardes, Elouan Gardes and Bojanowski, Piotr and Touvron, Hugo and Doron, Michael and Chen, Zitong S. and Moshkov, Nikita and Caron, Mathilde and Joulin, Armand and Pernice, Wolfgang M. and Caicedo, Juan C.}, journal={in review to PloS One on Computational Biology}, year={2025} } ``` --- ### README CHANNEL ADAPTIVE DINO (`docs/README_CHANNEL_ADAPTIVE_DINO.md`) # Scaling Channel-Adaptive Self-Supervised Learning [[`Paper `](https://openreview.net/forum?id=pT8sgtRVAf))] [[`BibTeX`](#citing-channeladaptivedino-and-dinov2)] **[Meta AI Research, FAIR](https://ai.facebook.com/research/)** Alice V. De Lorenci, Seungeun Yi, Théo Moutakanni, Piotr Bojanowski, Camille Couprie, Juan C. Caicedo, Wolfgang M. Pernice, with special thanks to Elouan Gardes for his contributions to the codebase. PyTorch implementation and pretrained model for ChannelAdaptive-DINO. The contents of this repo, including the code and model weights, are intended for research use only. It is not for use in medical procedures, including any diagnostics, treatment, or curative applications. Do not use this model for any clinical purpose or as a substitute for professional medical judgement. ## Pretrained model You can download the model weights trained on the Extended CHAMMI dataset (combination of five cell microscopy datasets with variable numbers of channels) on torchhub. ## Installation Follow instructions in the DINOv2 README. There are two additionnal dependencies to pandas and tifffile. ## What is included / not included This repository includes the Bag of Channel implementation, not the Hierarchical attention approach. ## Data preparation The CHAMMI dataset is available [here](https://github.com/chaudatascience/channel_adaptive_models). The HPA-FoV dataset is available [here](https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD2443) Content: a directory new_512_whole_images and two csv files: "whole_images_512_test.csv" "whole_images_512_train.csv" :warning: To execute the commands provided in the next sections for training and evaluation, the `dinov2` package should be included in the Python module search path, i.e. simply prefix the command to run with `PYTHONPATH=.`. ## Training ### Fast setup: training Channel-Adaptive DINO ViT-L/16 on HPA single cell dataset Run Channel-Adaptive DINO training on 4 A100-80GB nodes (32 GPUs) in a SLURM cluster environment with submitit: ```shell python dinov2/run/train/train.py \ --nodes 4 \ --config-file dinov2/configs/train/cell_dino/vitl16_boc_hpafov.yaml \ --output-dir \ train.dataset_path=HPAFoV:split=TRAIN:root=:wildcard=SEPARATE_CHANNELS" ``` Training time is approximately 2 days. The training code saves the weights of the teacher in the `eval` folder every 12500 iterations for evaluation. This example only performs pretraining on the HPA-FoV dataset. ## Evaluation The training code regularly saves the teacher weights. In order to evaluate the model, run the following evaluation on a single node: ### Linear Evaluation on HPAFoV ```shell PYTHONPATH=.:dinov2/data python dinov2/run/eval/cell_dino/linear.py \ --config-file dinov2/configs/eval/cell_dino/vitl16_channel_adaptive_pretrain.yaml \ --pretrained-weights /eval/training_359999/teacher_checkpoint.pth \ --output-dir /eval/training_359999/linear \ --train-dataset HPAFoV:split=TRAIN:mode=PROTEIN_LOCALIZATION:root= \ --val-dataset HPAFoV:split=VAL:mode=PROTEIN_LOCALIZATION:root= \ --val-metric-type mean_per_class_multilabel_f1 \ --loss-type binary_cross_entropy \ --bag-of-channels \ --crop-size 384 \ --n-last-blocks 4 \ --batch-size 32 \ --epoch-length 145 \ --epochs 30 \ --avgpool \ ``` ### KNN classification on CHAMMI Go to the docs directory, modifify some paths in launcher_knn_eval_on_chammi.sh and run ```shell ./launcher_knn_eval_on_chammi.sh WTC TASK_ONE ; ./launcher_knn_eval_on_chammi.sh WTC TASK_TWO ; ./launcher_knn_eval_on_chammi.sh HPA TASK_ONE ; ./launcher_knn_eval_on_chammi.sh HPA TASK_TWO ; ./launcher_knn_eval_on_chammi.sh HPA TASK_THREE ; ./launcher_knn_eval_on_chammi.sh CP TASK_ONE ; ./launcher_knn_eval_on_chammi.sh CP TASK_TWO ; ./launcher_knn_eval_on_chammi.sh CP TASK_THREE ; ./launcher_knn_eval_on_chammi.sh CP TASK_FOUR ; ``` ### Linear classification on CHAMMI Go to the docs directory, modifify some paths in launcher_CHAMMI_eval.sh and run ```shell ./launcher_CHAMMI_eval.sh WTC TASK_ONE ; ./launcher_CHAMMI_eval.sh WTC TASK_TWO ; ./launcher_CHAMMI_eval.sh HPA TASK_ONE ; ./launcher_CHAMMI_eval.sh HPA TASK_TWO ; ./launcher_CHAMMI_eval.sh HPA TASK_THREE ; ./launcher_CHAMMI_eval.sh CP TASK_ONE ; ./launcher_CHAMMI_eval.sh CP TASK_TWO ; ./launcher_CHAMMI_eval.sh CP TASK_THREE ; ./launcher_CHAMMI_eval.sh CP TASK_FOUR ; ``` | | WTC - Task 1 | WTC - Task 2 | HPA - Task 1 | HPA - Task 2 | HPA - Task 3 | CP - Task 1 | CP - Task 2 | CP - Task 3 | CP - Task 4 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | knn reproduced | 80.3 | 79.3 | 91.6 | 61.4 | 29.0 | 89.8 | 57.6 | 23.4 | 18.4 | | knn paper | 79.4 | 79.0 | 86.6 | 59.3 | 29.6 | 92.6 | 57.6 | 22.1 | 18.5 | | Linear reproduced | 89.9 | 87.9 | 92.7 | 87.2 | 66.2 | 89.9 | 59.8 | 26.6 | 32.5| | Linear paper | 90.5 | 89.2 | 88.3 | 84.7 | 65.0 | 90.5 | 60.5 | 25.8 | 32.7| ## License Cell-DINO code is released under the CC by NC licence See [LICENSE_CELL_DINO_CODE](LICENSE_CELL_DINO_CODE) for additional details. Model weights will be released under the FAIR Non-Commercial Research License. See [LICENSE_CELL_DINO_MODELS](LICENSE_CELL_DINO_MODELS) for additional details. ## Contributing See [contributing](CONTRIBUTING.md) and the [code of conduct](CODE_OF_CONDUCT.md). ## Citing ChannelAdaptiveDINO and DINOv2 If you find this repository useful, please consider giving a star :star: and citation :t-rex:: ``` @misc{Delorenci2025scaling, title={Scaling Channel-Adaptive Self-Supervised Learning}, author={V. De Lorenci, Alice and Yi, Seungeun and Moutakanni, Theo and Bojanowski, Piotr and Couprie, Camille and Caicedo, Juan C. and Pernice, Wolfgang M.}, journal={TMLR}, year={2025} } ``` ``` @misc{oquab2023dinov2, title={DINOv2: Learning Robust Visual Features without Supervision}, author={Oquab, Maxime and Darcet, Timothée and Moutakanni, Theo and Vo, Huy V. and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaaeldin and Howes, Russell and Huang, Po-Yao and Xu, Hu and Sharma, Vasu and Li, Shang-Wen and Galuba, Wojciech and Rabbat, Mike and Assran, Mido and Ballas, Nicolas and Synnaeve, Gabriel and Misra, Ishan and Jegou, Herve and Mairal, Julien and Labatut, Patrick and Joulin, Armand and Bojanowski, Piotr}, journal={arXiv:2304.07193}, year={2023} } ``` --- --- METRICS --- - Files Extracted: 3 - Estimated Token Budget: ~11571 tokens - Canonical Reference: https://codewiki.google/github.com/facebookresearch/dinov2