## File: README.md # OpenCLIP [[Paper]](https://arxiv.org/abs/2212.07143) [[Citations]](#citing) [[Clip Colab]](https://colab.research.google.com/github/mlfoundations/open_clip/blob/master/docs/Interacting_with_open_clip.ipynb) [[Coca Colab]](https://colab.research.google.com/github/mlfoundations/open_clip/blob/master/docs/Interacting_with_open_coca.ipynb) [](https://pypi.python.org/pypi/open_clip_torch) > ⚠️ **Main branch training stack notice** > > `main` now uses the post-refactor training stack by default. Training is organized around `TrainingTask` wrappers, dict-based batches, FSDP2 support, NaFlex image/audio pipelines, and multiple `torch.compile` strategies. The scope has grown well beyond the original refactor and now includes several new model families. **For the older release-stable training API, pin to the [`v3` branch](https://github.com/mlfoundations/open_clip/tree/v3) or the latest 3.x release on PyPI.** Inference usage of pretrained image/text models is still intended to be compatible, but training scripts and downstream integrations should review the changes below before upgrading. > > **New / experimental model families on `main`:** > - **NaFlex CLIP** — variable-resolution/aspect image towers (timm `naflexvit`) with token-budget batching (`--use-naflex`, `naflex_*` configs) > - **NaFlex CLAP** — audio-text contrastive training with variable-duration audio (`naflexclap_*` configs) > - **NaFlex GenLIP / GenLAP** — generative image/audio captioning with prefix-LM attention and packed `[media ; text]` rows (`naflexgenlip_*`, `naflexgenlap_*` configs, tiktoken text) > - **Modern text tower** — `text_cfg.text_arch="modern"`: RoPE, SwiGLU/ReLU², RMSNorm, masked pooling (`eos`/`mean`/`map`), with optional qk-norm, gated attention, register tokens, sandwich norm, value residuals, and zero-init residual (`moderntext-*` configs) > - **Variable-length text** — `text_cfg.variable_text=true` pads captions to the per-batch max instead of a fixed context length (works with the modern tower and HF towers) > - **Hugging Face ModernBERT text towers** — e.g. `gte-modernbert-base-ViT-B-32-256` > - **MaMMUT** — single text decoder used in two passes per [the paper](https://arxiv.org/abs/2303.16839): bi-directional without cross-attention for contrastive learning, causal with cross-attention for captioning (trains via the CoCa task/loss). Two config families, mirroring coca/coca2: `mammut_*` reproduces the original LAION-fork numerics exactly via legacy flags (`pool_type="avg_all"`, `use_pad_mask=false`) and loads the released [LAION openMaMMUT-ViT-L-14](https://huggingface.co/laion/openMaMMUT-ViT-L-14-DataComp-1.4B-s12.8B-b180K) weights (pretrained tag `datacomp1b_s12_8b_b180k`, or directly via `hf-hub:` — fork-format configs and state dicts are translated on load); `mammut2_*` uses the corrected defaults (masked-mean text pooling, pad masking). A modern-arch decoder variant is available via `multimodal_cfg.text_arch="modern"` (`mammut2-moderntext_*`) > - **CoCa v2 configs** — `coca2_*`: paper-faithful attentional pooling (`vision_cfg.attentional_pool="cascade"`, the paper's default — a separate single-query contrastive pooler, so the caption decoder now cross-attends over all 256 generative pooler tokens instead of 255, fixing [#458](https://github.com/mlfoundations/open_clip/issues/458)) and the corrected CLS/pad attention mask (`text_cfg.correct_cls_mask=true`), per [#554](https://github.com/mlfoundations/open_clip/issues/554). A modern-text variant pairs the modern tower with a modern multimodal decoder (`coca2-moderntext_*`). Existing `coca_*` configs and released weights are unchanged > - **Text validity masks for generative models** — CoCa/MaMMUT `forward()`/`encode_text()` accept `text_valid` (`[B, L]`, True = real token); tokenizers can emit exact masks (`tokenizer(texts, output_mask=True)`), and caption labels are masked to `-100` from them. Fixes the SimpleTokenizer pad-collision class (id 0 is a real token, `'!'` merges like `x!=y` emit it mid-caption); absent a mask, behavior falls back to the historical `text != pad_id` derivation. Text towers keep the HF-style `attention_mask` kwarg at the tower boundary > > **Breaking changes — training CLI:** > - `--horovod` removed (Horovod support deleted; DDP/FSDP2 only) > - `--torchscript` and `--trace` removed (`torch.jit` is being deprecated upstream) > - Default `--precision` changed from `amp` → `amp_bf16` (silent behavior change; pass `--precision amp` explicitly to keep fp16 AMP) > - `--naflex-max-tokens-per-batch` now defaults to unset. The local token budget is inferred as `--batch-size * max(--naflex-seq-lens)`; GenLIP/GenLAP also include their caption-token cap in the per-row cost. Pass an explicit token budget to preserve older runs that relied on the previous `16384` default. > > **New training CLI flags (opt-in):** > - `--fsdp` — use FSDP2 (`fully_shard`) instead of DDP > - `--fsdp-no-reshard-after-forward`, `--fsdp-offload-cpu` > - `--fsdp-checkpoint {full,sharded}` — full gathers to rank-0 as a single `.pt`; `sharded` uses DCP per-rank shards (faster, lower memory) > - `--torchcompile-strategy {task,model,step}` — choose whether `torch.compile` captures task forward/loss, the underlying model, or the full single-batch train step > - `--use-naflex` and `--naflex-*` flags — enable NaFlex variable-aspect image pipelines for compatible timm/OpenCLIP ViT-family models (token-budget batching via `--naflex-seq-lens` / `--naflex-max-tokens-per-batch`; the same machinery drives the NaFlex audio and generative models) > - `--audio-*` and `--audio-zeroshot-*` flags — enable CLAP audio preprocessing/training and Hugging Face audio zero-shot evaluation > - `--length-bucketing`, `--bucket-pool`, `--bucket-chunk` — reorder the train stream by sample length (caption and/or audio tokens) to tighten per-batch padding; the bucket pool holds raw, undecoded samples > - `--text-pad-multiple` — round per-batch variable-text length up to a multiple, bounding the distinct sequence lengths `torch.compile` sees (text-axis analogue of `--naflex-pad-multiple`) > - `--text-attention-mask` — emit a per-sample text validity mask (batch key `text_valid`) from the tokenizer, consumed by CoCa/MaMMUT for attention/pooling and `-100` caption-label masking. Default auto-enables for CoCa/MaMMUT (except under `--distill`) and is rejected for tasks that don't consume it > - `--caption-z-loss-weight`, `--caption-loss-compute-dtype {float32,model}`, and `--caption-loss-chunk-size` — configure the next-token objective shared by CoCa/MaMMUT and GenLIP/GenLAP. Defaults preserve the existing fp32 CE with no z-loss; `model` preserves the loss-logit dtype and ambient autocast policy while returned loss/component scalars remain fp32 > > **Breaking changes — Python API:** > - `trace_model` removed from the top-level `open_clip` namespace > - `load_openai_model`, `list_openai_models`, and `build_model_from_openai_state_dict` removed. Original-OpenAI weights are still loadable through the standard `create_model_from_pretrained(..., pretrained='openai')` path, which now routes through HuggingFace Hub (`timm/*_clip.openai`) instead of `torch.jit.load` on `openaipublic.azureedge.net` archives. Removing the JIT path closes an arbitrary-code-execution surface (JIT archives can ship pickled code). > - Training pipeline wraps `model + loss` in a `TrainingTask` subclass (`CLIPTask`, `SigLIPTask`, `CoCaTask`, `DistillCLIPTask`, `CLAPTask`). Code that previously called `train_one_epoch(model, loss, ...)` or `evaluate(model, ...)` directly should switch to passing a task. `create_loss(args)` remains available as a standalone loss factory for non-task training loops. > - Data pipelines emit dict batches instead of tuples. Image/text loaders use `{"image": ..., "text": ...}`; CLAP audio loaders use `{"audio": ..., "text": ...}`. Tuple-style image/text calls through `task(images, texts)` still work via a backward-compat path, but downstream code that iterates a dataloader directly should read the named keys. > - CoCa's autoregressive label shift moved out of `coca_model.py` and into `CoCaTask`. `coca_model.forward()` no longer performs the `[:, :-1]` / `[:, 1:]` slicing — callers that relied on that behavior outside training should handle labels themselves. > - CoCa's exact-mask API adds `text_valid` after `text` in `encode_text`, `forward`, and `forward_intermediates`. This shifts the older trailing positional arguments (`normalize`, `image_latent`, `image_indices`, and so on); pass those arguments by keyword. For example, replace `model.encode_text(text, False)` with `model.encode_text(text, normalize=False)`. > - `CLIPTextCfg.eos_id` no longer defaults to `2` (that value is only correct for XLM-style vocabs). Configs using `pool_type="eos"` must set `eos_id` explicitly, and `get_tokenizer` now validates `eos_id`/`pad_id` against the resolved tokenizer, raising on mismatch instead of pooling/masking silently wrong positions. > - `HFTokenizer` no longer fabricates `pad_token_id=0` when the underlying tokenizer has no pad token (id 0 is a real token in most BPE vocabs); variable-text setups fail fast instead. It also forces `padding_side='right'`, which all OpenCLIP pooling/masking assumes. > - `MaxPooler` (`hf_pooler_type="max_pooler"`) mask polarity fixed — it previously max-pooled over the padding positions instead of the valid ones. > - `CoCa.__init__` no longer takes a `pad_id` argument — `model.pad_id` is derived from the text tower (the id it actually masks with: `text_cfg.pad_id` for native towers, the transformers config pad for HF towers). MaMMUT follows the same pattern. This fixes `coca_roberta-*`, which previously masked with roberta's pad (1) in the tower while the loss ignored 0 — its config now declares `pad_id: 1` and the caption loss no longer trains on padding. > - `CoCaTask` builds caption labels masked to `-100`; `CoCaLoss`'s cross-entropy uses `ignore_index=-100`. Its `pad_id` arg is retained for standalone callers passing raw labels (default `0` preserves the old value-based behavior; the task path passes `None`). Validation generative-loss metrics are likewise mask/pad-aware and will report different (correct) values for nonzero-pad models. > - `create_task()` and `create_loss(args, model=...)` dispatch on the built (unwrapped) model type instead of matching `args.model` substrings — `hf-hub:`/`local-dir:`/renamed configs previously fell through to `CLIPTask` and silently dropped the caption loss. Name-based fallback remains only in `create_loss` when no model is passed. > - `--lock-image` now freezes vision attentional poolers (they were previously left trainable when locking CoCa-style towers, and were ungrouped under `--image-layer-decay`). > - `MultimodalTransformer` (the CoCa text decoder) now initializes its parameters — previously its `init_parameters` was never called and `text_projection` was uninitialized memory on fresh builds. New training runs won't bit-match runs started under older code; checkpoint loading is unaffected. `generate()` for CoCa/MaMMUT now derives `pad`/`bos`/`eos` ids from the model/tower/HF config instead of hardcoding CLIP token ids (fixes generation for HF-tower CoCa), and shares one implementation in `open_clip.generation`. > - `get_tokenizer` validation is stricter for generative configs (CoCa/MaMMUT/GenLIP): an explicit nonzero `pad_id` with a tokenizer that reserves no pad token raises, and an unset `pad_id` against a reserved nonzero tokenizer pad warns (the pad-value fallback would train the caption loss on padding). > - WebDataset pipelines now assemble as `tokenize -> [length bucketing] -> decode -> transform` so the bucket pool holds raw bytes instead of decoded images/waveforms (10-50x less dataloader-worker memory). The old decode-first assembly is preserved in `open_clip_train.legacy_data` (no bucketing/NaFlex) and used by `legacy_main`. > > **Dependency bump:** Minimum `torch>=2.6` (was `>=2.0`). This is the version where `torch.load(weights_only=True)` became the default — all checkpoint loads in this repo now pass `weights_only=True` explicitly with no `weights_only=False` fallback. If you're resuming training with a custom optimizer that pickles non-allowlisted Python types, register them via `torch.serialization.add_safe_globals([...])` before calling `load_checkpoint`. > > **Checkpoint compatibility:** Existing pretrained `.pt` checkpoints load without changes. Training checkpoints saved on `main` include a `state_dict` key that's compatible with prior versions; EMA, optimizer state, and optional training counters are also preserved. `0-D` vs `1-D` scalar reshape from the FSDP path is reconciled on load, so you can resume a DDP-trained checkpoint under FSDP2 and vice versa. > > **Legacy training entry point:** `python -m open_clip_train.legacy_main` remains available for older image/text training scripts that need the pre-task loop, and pairs with the frozen decode-first data pipelines in `open_clip_train.legacy_data`. It does not support the full task-era feature set (for example FSDP2, EMA, CLAP audio training, NaFlex, length bucketing, or the task/step `torch.compile` integration), and should be treated as a compatibility shim rather than the path for new training work. Welcome to an open source implementation of OpenAI's [CLIP](https://arxiv.org/abs/2103.00020) (Contrastive Language-Image Pre-training). Using this codebase, we have trained several models on a variety of data sources and compute budgets, ranging from [small-scale experiments](docs/LOW_ACC.md) to larger runs including models trained on datasets such as [LAION-400M](https://arxiv.org/abs/2111.02114), [LAION-2B](https://arxiv.org/abs/2210.08402) and [DataComp-1B](https://arxiv.org/abs/2304.14108). Many of our models and their scaling properties are studied in detail in the paper [reproducible scaling laws for contrastive language-image learning](https://arxiv.org/abs/2212.07143). Some of the best models we've trained and their zero-shot ImageNet-1k accuracy are shown below, along with the ViT-L model trained by OpenAI and other state-of-the-art open source alternatives (all can be loaded via OpenCLIP). We provide more details about our full collection of pretrained models [here](docs/PRETRAINED.md), and zero-shot results for 38 datasets [here](docs/openclip_results.csv). | Model | Training data | Resolution | # of samples seen | ImageNet zero-shot acc. | | -------- | ------- | ------- | ------- | ------- | | ConvNext-Base | LAION-2B | 256px | 13B | 71.5% | | ConvNext-Large | LAION-2B | 320px | 29B | 76.9% | | ConvNext-XXLarge | LAION-2B | 256px | 34B | 79.5% | | ViT-B-32-256 | DataComp-1B | 256px | 34B | 72.8% | | ViT-B-16 | DataComp-1B | 224px | 13B | 73.5% | | ViT-L-14 | LAION-2B | 224px | 32B | 75.3% | | ViT-H-14 | LAION-2B | 224px | 32B | 78.0% | | ViT-L-14 | DataComp-1B | 224px | 13B | 79.2% | | ViT-bigG-14 | LAION-2B | 224px | 34B | 80.1% | | | | | | | | ViT-L-14-quickgelu [(Original CLIP)](https://arxiv.org/abs/2103.00020) | WIT | 224px | 13B | 75.5% | | ViT-SO400M-14-SigLIP [(SigLIP)](https://arxiv.org/abs/2303.15343) | WebLI | 224px | 45B | 82.0% | | ViT-L-14 [(DFN)](https://arxiv.org/abs/2309.17425) | DFN-2B | 224px | 39B | 82.2% | | ViT-L-16-256 [(SigLIP2)](https://arxiv.org/abs/2502.14786) | WebLI (multi-lang) | 256px | 40B | 82.5% | | ViT-SO400M-14-SigLIP-384 [(SigLIP)](https://arxiv.org/abs/2303.15343) | WebLI | 384px | 45B | 83.1% | | ViT-H-14-quickgelu [(DFN)](https://arxiv.org/abs/2309.17425) | DFN-5B | 224px | 39B | 83.4% | | PE-Core-L-14-336 [(PE)](https://arxiv.org/abs/2504.13181) | MetaCLIP-5.4B | 336px | 58B | 83.5% | | ViT-SO400M-16-SigLIP2-384 [(SigLIP2)](https://arxiv.org/abs/2502.14786) | WebLI (multi-lang) | 384px | 40B | 84.1% | | ViT-H-14-378-quickgelu [(DFN)](https://arxiv.org/abs/2309.17425) | DFN-5B | 378px | 44B | 84.4% | | ViT-gopt-16-SigLIP2-384 [(SigLIP2)](https://arxiv.org/abs/2502.14786) | WebLI (multi-lang) | 384px | 40B | 85.0% | | PE-Core-bigG-14-448 [(PE)](https://arxiv.org/abs/2504.13181) | MetaCLIP-5.4B | 448px | 86B | 85.4% | Model cards with additional model specific details can be found on the Hugging Face Hub under the OpenCLIP library tag: https://huggingface.co/models?library=open_clip. If you found this repository useful, please consider [citing](#citing). We welcome anyone to submit an issue or send an email if you have any other requests or suggestions. Note that portions of `src/open_clip/` modelling and tokenizer code are adaptations of OpenAI's official [repository](https://github.com/openai/CLIP). ## Approach | | |:--:| | Image Credit: https://github.com/openai/CLIP | ## Usage ``` pip install open_clip_torch ``` ```python import torch from PIL import Image import open_clip model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k') model.eval() # model in train mode by default, impacts some models with BatchNorm or stochastic depth active tokenizer = open_clip.get_tokenizer('ViT-B-32') image = preprocess(Image.open("docs/CLIP.png")).unsqueeze(0) text = tokenizer(["a diagram", "a dog", "a cat"]) with torch.no_grad(), torch.autocast("cuda"): image_features = model.encode_image(image) text_features = model.encode_text(text) image_features /= image_features.norm(dim=-1, keepdim=True) text_features /= text_features.norm(dim=-1, keepdim=True) text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1) print("Label probs:", text_probs) # prints: [[1., 0., 0.]] ``` If model uses `timm` image encoders (convnext, siglip, eva, etc) ensure the latest timm is installed. Upgrade `timm` if you see 'Unknown model' errors for the image encoder. If model uses transformers tokenizers, ensure `transformers` is installed. See also this [[Clip Colab]](https://colab.research.google.com/github/mlfoundations/open_clip/blob/master/docs/Interacting_with_open_clip.ipynb). To compute billions of embeddings efficiently, you can use [clip-retrieval](https://github.com/rom1504/clip-retrieval) which has openclip support. ### Pretrained models We offer a simple model interface to instantiate both pre-trained and untrained models. To see which pretrained models are available, use the following code snippet. More details about our pretrained models are available [here](docs/PRETRAINED.md). ```python >>> import open_clip >>> open_clip.list_pretrained() ``` You can find more about the models we support (e.g. number of parameters, FLOPs) in [this table](docs/model_profile.csv). NOTE: Many existing checkpoints use the QuickGELU activation from the original OpenAI models. This activation is actually less efficient than native torch.nn.GELU in recent versions of PyTorch. The model defaults are now nn.GELU, so one should use model definitions with `-quickgelu` postfix for the OpenCLIP pretrained weights. All OpenAI pretrained weights will always default to QuickGELU. One can also use the non `-quickgelu` model definitions with pretrained weights using QuickGELU but there will be an accuracy drop, for fine-tune that will likely vanish for longer runs. Future trained models will use nn.GELU. ### Loading models Models can be loaded with `open_clip.create_model_and_transforms`, as shown in the example below. The model name and corresponding `pretrained` keys are compatible with the outputs of `open_clip.list_pretrained()`. The `pretrained` argument also accepts local paths, for example `/path/to/my/b32.pt`. You can also load checkpoints from huggingface this way. To do so, download the `open_clip_pytorch_model.bin` file (for example, [https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K/tree/main](https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K/blob/main/open_clip_pytorch_model.bin)), and use `pretrained=/path/to/open_clip_pytorch_model.bin`. ```python # pretrained also accepts local paths model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k') ``` ## Fine-tuning on classification tasks This repository is focused on training CLIP models. To fine-tune a *trained* zero-shot model on a downstream classification task such as ImageNet, please see [our other repository: WiSE-FT](https://github.com/mlfoundations/wise-ft). The [WiSE-FT repository](https://github.com/mlfoundations/wise-ft) contains code for our paper on [Robust Fine-tuning of Zero-shot Models](https://arxiv.org/abs/2109.01903), in which we introduce a technique for fine-tuning zero-shot models while preserving robustness under distribution shift. ## Data To download datasets as webdataset, we recommend [img2dataset](https://github.com/rom1504/img2dataset). ### Conceptual Captions See [cc3m img2dataset example](https://github.com/rom1504/img2dataset/blob/main/dataset_examples/cc3m.md). ### YFCC and other datasets In addition to specifying the training data via CSV files as mentioned above, our codebase also supports [webdataset](https://github.com/webdataset/webdataset), which is recommended for larger scale datasets. The expected format is a series of `.tar` files. Each of these `.tar` files should contain two files for each training example, one for the image and one for the corresponding text. Both files should have the same name but different extensions. For instance, `shard_001.tar` could contain files such as `abc.jpg` and `abc.txt`. You can learn more about `webdataset` at [https://github.com/webdataset/webdataset](https://github.com/webdataset/webdataset). We use `.tar` files with 1,000 data points each, which we create using [tarp](https://github.com/webdataset/tarp). You can download the YFCC dataset from [Multimedia Commons](http://mmcommons.org/). Similar to OpenAI, we used a subset of YFCC to reach the aforementioned accuracy numbers. The indices of images in this subset are in [OpenAI's CLIP repository](https://github.com/openai/CLIP/blob/main/data/yfcc100m.md). ## Training CLIP ### Install We advise you first create a virtual environment with: ``` python3 -m venv .env source .env/bin/activate pip install -U pip ``` You can then install openclip for training with `pip install 'open_clip_torch[training]'`. #### Development If you want to make changes to contribute code, you can clone openclip then run `make install` in openclip folder (after creating a virtualenv) Install pip PyTorch as per https://pytorch.org/get-started/locally/ You may run `make install-training` to install training deps #### Testing Test can be run with `make install-test` then `make test` `python -m pytest -x -s -v tests -k "training"` to run a specific test Running regression tests against a specific git revision or tag: 1. Generate testing data ```sh python tests/util_test.py --model RN50 RN101 --save_model_list models.txt --git_revision 9d31b2ec4df6d8228f370ff20c8267ec6ba39383 ``` **_WARNING_: This will invoke git and modify your working tree, but will reset it to the current state after data has been generated! \ Don't modify your working tree while test data is being generated this way.** 2. Run regression tests ```sh OPEN_CLIP_TEST_REG_MODELS=models.txt python -m pytest -x -s -v -m regression_test ``` ### Sample single-process running code: ```bash python -m open_clip_train.main \ --save-frequency 1 \ --zeroshot-frequency 1 \ --report-to tensorboard \ --train-data="/path/to/train_data.csv" \ --val-data="/path/to/validation_data.csv" \ --csv-img-key filepath \ --csv-caption-key title \ --imagenet-val=/path/to/imagenet/root/val/ \ --warmup 10000 \ --batch-size=128 \ --lr=1e-3 \ --wd=0.1 \ --epochs=30 \ --workers=8 \ --model RN50 ``` Note: `imagenet-val` is the path to the *validation* set of ImageNet for zero-shot evaluation, not the training set! You can remove this argument if you do not want to perform zero-shot evaluation on ImageNet throughout training. Note that the `val` folder should contain subfolders. If it does not, please use [this script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh). ### torch.compile Training supports `torch.compile` via `--torchcompile`. Use `--torchcompile-strategy task` for the common case: it compiles the task train/eval forward callables while keeping the `TrainingTask` object itself unwrapped for checkpointing, batch preparation, eval, and zero-shot helpers. Use `--torchcompile-strategy model` when you want the lower-risk pre-DDP behavior of compiling only the underlying trainable module, especially for fp16 AMP runs that need a `GradScaler`. Use `--torchcompile-strategy step` for the most aggressive single-batch path: it compiles forward, loss, backward, optional grad clipping, optimizer step, and logit-scale clamp, but requires `--accum-freq 1` and a precision mode without `GradScaler` such as the default `amp_bf16` or `fp32`. `--torchcompile-backend` and `--torchcompile-mode` are passed through to `torch.compile`. ### Multi-GPU and Beyond This code has been battle tested up to 1024 A100s and offers a variety of solutions for distributed training. We include native support for SLURM clusters. As the number of devices used to train increases, so does the space complexity of the the logit matrix. Using a naïve all-gather scheme, space complexity will be `O(n^2)`. Instead, complexity may become effectively linear if the flags `--gather-with-grad` and `--local-loss` are used. This alteration results in one-to-one numerical results as the naïve method. #### Epochs For larger datasets (eg Laion2B), we recommend setting `--train-num-samples` to a lower value than the full epoch, for example `--train-num-samples 135646078` to 1/16 of an epoch in conjunction with `--dataset-resampled` to do sampling with replacement. This allows having frequent checkpoints to evaluate more often. #### Patch Dropout [Recent research](https://arxiv.org/abs/2212.00794) has shown that one can dropout half to three-quarters of the visual tokens, leading to up to 2-3x training speeds without loss of accuracy. You can set this on your visual transformer config with the key `patch_dropout`. In the paper, they also finetuned without the patch dropout at the end. You can do this with the command-line argument `--force-patch-dropout 0.` #### Multiple data sources OpenCLIP supports using multiple data sources, by separating different data paths with `::`. For instance, to train on CC12M and on LAION, one might use `--train-data "/data/cc12m/cc12m-train-{0000..2175}.tar::/data/LAION-400M/{00000..41455}.tar"`. Using `--dataset-resampled` is recommended for these cases. By default, on expectation the amount of times the model will see a sample from each source is proportional to the size of the source. For instance, when training on one data source with size 400M and one with size 10M, samples from the first source are 40x more likely to be seen in expectation. We also support different weighting of the data sources, by using the `--train-data-upsampling-factors` flag. For instance, using `--train-data-upsampling-factors=1::1` in the above scenario is equivalent to not using the flag, and `--train-data-upsampling-factors=1::2` is equivalent to upsampling the second data source twice. If you want to sample from data sources with the same frequency, the upsampling factors should be inversely proportional to the sizes of the data sources. For instance, if dataset `A` has 1000 samples and dataset `B` has 100 samples, you can use `--train-data-upsampling-factors=0.001::0.01` (or analogously, `--train-data-upsampling-factors=1::10`). #### Single-Node We make use of `torchrun` to launch distributed jobs. The following launches a a job on a node of 4 GPUs: ```bash cd open_clip/src torchrun --nproc_per_node 4 -m open_clip_train.main \ --train-data '/data/cc12m/cc12m-train-{0000..2175}.tar' \ --train-num-samples 10968539 \ --dataset-type webdataset \ --batch-size 320 \ --precision amp \ --workers 4 \ --imagenet-val /data/imagenet/validation/ ``` #### Multi-Node The same script above works, so long as users include information about the number of nodes and host node. ```bash cd open_clip/src torchrun --nproc_per_node=4 \ --rdzv_endpoint=$HOSTE_NODE_ADDR \ -m open_clip_train.main \ --train-data '/data/cc12m/cc12m-train-{0000..2175}.tar' \ --train-num-samples 10968539 \ --dataset-type webdataset \ --batch-size 320 \ --precision amp \ --workers 4 \ --imagenet-val /data/imagenet/validation/ ``` #### SLURM This is likely the easiest solution to utilize. The following script was used to train our largest models: ```bash #!/bin/bash -x #SBATCH --nodes=32 #SBATCH --gres=gpu:4 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=6 #SBATCH --wait-all-nodes=1 #SBATCH --job-name=open_clip #SBATCH --account=ACCOUNT_NAME #SBATCH --partition PARTITION_NAME eval "$(/path/to/conda/bin/conda shell.bash hook)" # init conda conda activate open_clip export CUDA_VISIBLE_DEVICES=0,1,2,3 export MASTER_PORT=12802 master_addr=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1) export MASTER_ADDR=$master_addr cd /shared/open_clip export PYTHONPATH="$PYTHONPATH:$PWD/src" srun --cpu_bind=v --accel-bind=gn python -u src/open_clip_train/main.py \ --save-frequency 1 \ --report-to tensorboard \ --train-data="/data/LAION-400M/{00000..41455}.tar" \ --warmup 2000 \ --batch-size=256 \ --epochs=32 \ --workers=8 \ --model ViT-B-32 \ --name "ViT-B-32-Vanilla" \ --seed 0 \ --local-loss \ --gather-with-grad ``` ### Resuming from a checkpoint: ```bash python -m open_clip_train.main \ --train-data="/path/to/train_data.csv" \ --val-data="/path/to/validation_data.csv" \ --resume /path/to/checkpoints/epoch_K.pt ``` ### Training CoCa: Training [CoCa](https://arxiv.org/abs/2205.01917) models is enabled through specifying a CoCa config using the ```--model``` parameter of the training script. Currently available configs are "coca_base", "coca_ViT-B-32", and "coca_roberta-ViT-B-32" (which uses RoBERTa as the text encoder). CoCa configs are different from CLIP configs because they have an additional "multimodal_cfg" component which specifies parameters for the multimodal text decoder. Here's an example from the coca_ViT-B-32 config: ```json "multimodal_cfg": { "context_length": 76, "vocab_size": 49408, "width": 512, "heads": 8, "layers": 12, "latent_dim": 512, "attn_pooler_heads": 8 } ``` Credit to [lucidrains](https://github.com/lucidrains) for [initial code](https://github.com/lucidrains/CoCa-pytorch), [gpucce](https://github.com/gpucce) for adapting the code to open_clip, and [iejMac](https://github.com/iejMac) for training the models. ### Generating text with CoCa ```python import open_clip import torch from PIL import Image model, _, transform = open_clip.create_model_and_transforms( model_name="coca_ViT-L-14", pretrained="mscoco_finetuned_laion2B-s13B-b90k" ) im = Image.open("cat.jpg").convert("RGB") im = transform(im).unsqueeze(0) with torch.no_grad(), torch.cuda.amp.autocast(): generated = model.generate(im) print(open_clip.decode(generated[0]).split("")[0].replace("", "")) ``` See also this [[Coca Colab]](https://colab.research.google.com/github/mlfoundations/open_clip/blob/master/docs/Interacting_with_open_coca.ipynb) ### Fine Tuning CoCa To fine-tune coca on mscoco, first create the dataset, one way is using a csvdataset and perhaps the simplest way to do it is using [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark) which in turn uses [pycocotools](https://github.com/cocodataset/cocoapi) (that can be used also by itself). ```python from clip_benchmark.datasets.builder import build_dataset import pandas as pd import os root_path = "path/to/data/dir" # set this to smth meaningful ds = build_dataset("mscoco_captions", root=root_path, split="train", task="captioning") # this downloads the dataset if it is not there already coco = ds.coco imgs = coco.loadImgs(coco.getImgIds()) future_df = {"filepath":[], "title":[]} for img in imgs: caps = coco.imgToAnns[img["id"]] for cap in caps: future_df["filepath"].append(img["file_name"]) future_df["title"].append(cap["caption"]) pd.DataFrame.from_dict(future_df).to_csv( os.path.join(root_path, "train2014.csv"), index=False, sep="\t" ) ``` This should create a csv dataset that one can use to fine-tune coca with open_clip ```bash python -m open_clip_train.main \ --dataset-type "csv" \ --train-data "path/to/data/dir/train2014.csv" \ --warmup 1000 \ --batch-size 128 \ --lr 1e-5 \ --wd 0.1 \ --epochs 1 \ --workers 3 \ --model "coca_ViT-L-14" \ --report-to "wandb" \ --coca-contrastive-loss-weight 0 \ --coca-caption-loss-weight 1 \ --log-every-n-steps 100 ``` This is a general setting, open_clip has very parameters that can be set, ```python -m open_clip_train.main --help``` should show them. The only relevant change compared to pre-training are the two arguments ```bash --coca-contrastive-loss-weight 0 --coca-caption-loss-weight 1 ``` which make the model only train the generative side. ### Training with pre-trained language models as text encoder: If you wish to use different language models as the text encoder for CLIP you can do so by using one of the Hugging Face model configs in ```src/open_clip/model_configs``` and passing in it's tokenizer as the ```--model``` and ```--hf-tokenizer-name``` parameters respectively. Currently we only support RoBERTa ("test-roberta" config), however adding new models should be trivial. You can also determine how many layers, from the end, to leave unfrozen with the ```--lock-text-unlocked-layers``` parameter. Here's an example command to train CLIP with the RoBERTa LM that has it's last 10 layers unfrozen: ```bash python -m open_clip_train.main \ --train-data="pipe:aws s3 cp s3://s-mas/cc3m/{00000..00329}.tar -" \ --train-num-samples 3000000 \ --val-data="pipe:aws s3 cp s3://s-mas/cc3m/{00330..00331}.tar -" \ --val-num-samples 10000 \ --dataset-type webdataset \ --batch-size 256 \ --warmup 2000 \ --epochs 10 \ --lr 5e-4 \ --precision amp \ --workers 6 \ --model "roberta-ViT-B-32" \ --lock-text \ --lock-text-unlocked-layers 10 \ --name "10_unfrozen" \ --report-to "tensorboard" \ ``` ### Loss Curves When run on a machine with 8 GPUs the command should produce the following training curve for Conceptual Captions: More detailed curves for Conceptual Captions are given at [/docs/clip_conceptual_captions.md](/docs/clip_conceptual_captions.md). When training a RN50 on YFCC the same hyperparameters as above are used, with the exception of `lr=5e-4` and `epochs=32`. Note that to use another model, like `ViT-B/32` or `RN50x4` or `RN50x16` or `ViT-B/16`, specify with `--model RN50x4`. ### Logging For tensorboard logging, run: ```bash tensorboard --logdir=logs/tensorboard/ --port=7777 ``` For wandb logging, we recommend looking at the `step` variable instead of `Step`, since the later was not properly set in earlier versions of this codebase. For older runs with models trained before https://github.com/mlfoundations/open_clip/pull/613, the `Step` variable should be ignored. For newer runs, after that PR, the two variables are the same. ## Evaluation / Zero-Shot We recommend https://github.com/LAION-AI/CLIP_benchmark#how-to-use for systematic evaluation on 40 datasets. ### Evaluating local checkpoint: ```bash python -m open_clip_train.main \ --val-data="/path/to/validation_data.csv" \ --model RN101 \ --pretrained /path/to/checkpoints/epoch_K.pt ``` ### Evaluating hosted pretrained checkpoint on ImageNet zero-shot prediction: ```bash python -m open_clip_train.main \ --imagenet-val /path/to/imagenet/validation \ --model ViT-B-32-quickgelu \ --pretrained laion400m_e32 ``` ### Evaluating CLAP audio models on zero-shot audio classification: Audio zero-shot evaluation uses Hugging Face audio classification datasets. Install the optional audio dependencies before running these examples. ```bash pip install 'datasets[audio]' torchaudio torchlibrosa ``` ESC-50: ```bash python -m open_clip_train.main \ --model CLAP-HTSAT-tiny-Roberta-base-fused \ --pretrained laion \ --audio-zeroshot-dataset ashraq/esc50 \ --audio-zeroshot-split train \ --batch-size 64 \ --workers 4 \ --device cuda \ --zeroshot-frequency 1 ``` UrbanSound8K: ```bash python -m open_clip_train.main \ --model CLAP-HTSAT-tiny-Roberta-base-fused \ --pretrained laion \ --audio-zeroshot-dataset mteb/urbansound8K \ --audio-zeroshot-split train \ --audio-zeroshot-target-key classID \ --audio-zeroshot-class-key class \ --batch-size 64 \ --workers 4 \ --device cuda \ --zeroshot-frequency 1 ``` ### Model distillation You can distill from a pre-trained by using `--distill-model` and `--distill-pretrained` to specify the model you'd like to distill from. For instance, to distill from OpenAI ViT-L/14 use `--distill-model ViT-L-14 --distill-pretrained openai`. ### Gradient accumulation To simulate larger batches use `--accum-freq k`. If per gpu batch size, `--batch-size`, is `m`, then the effective batch size will be `k * m * num_gpus`. When increasing `--accum-freq` from its default of 1, samples/s will remain approximately constant (batch size will double, as will time-per-batch). It is recommended to use other features to reduce batch size such as `--grad-checkpointing --local-loss --gather-with-grad` before increasing `--accum-freq`. `--accum-freq` can be used in addition to these features. Instead of 1 forward pass per example, there are now 2 forward passes per-example. However, the first is done with `torch.no_grad`. There is some additional GPU memory required --- the features and data from all `m` batches are stored in memory. There are also `m` loss computations instead of the usual 1. For more information see Cui et al. (https://arxiv.org/abs/2112.09331) or Pham et al. (https://arxiv.org/abs/2111.10050). ### Int8 Support We have beta support for int8 inference via bitsandbytes' `Linear8bitLt` (the LLM.int8() layer). Only the MLP linear layers (`c_fc`/`c_proj`) are quantized by default; attention's combined QKV projection isn't an `nn.Linear` submodule and isn't covered yet (see the TODO in `src/open_clip/utils.py`). ```python from functools import partial import bitsandbytes as bnb import open_clip model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k') model = model.half() int8_linear_layer = partial(bnb.nn.Linear8bitLt, has_fp16_weights=False) int8_model = open_clip.utils.replace_linear(model, int8_linear_layer, include_modules=['c_fc', 'c_proj']).cuda() open_clip.utils.convert_int8_model_to_inference_mode(int8_model) # stamps the dtype bookkeeping the model needs post-quantization ``` **Accuracy / speed, measured, not estimated:** on ViT-B-32-quickgelu (`openai` weights), zero-shot classification on CIFAR-10 was 88.76% (fp16) vs. 88.83% (int8) — a **+0.07 point** difference, i.e. no meaningful accuracy cost for quantizing just the MLP layers post-hoc (no fine-tuning). This is a single-template, single-dataset, single-model measurement, not a comprehensive benchmark, but it's real and reproducible (see the tutorial notebook). Speed was slightly *worse* for int8 (53.9ms vs. 56.9ms for a batch of 128, about 5.6% slower) — at this model scale, the extra kernel launches int8 quantization/dequantization requires per forward pass outweigh any raw int8 tensor-core throughput advantage, since `Linear8bitLt` was designed around much larger (billion-parameter) models than CLIP's transformer blocks. The practical benefit here is **memory** (roughly a 2x reduction on the quantized layers' weight tensors), not speed or accuracy. **Saving and loading a quantized model:** `replace_linear` swaps `nn.Linear` submodules for bitsandbytes layers in place, so a `state_dict` saved from the quantized model only loads back correctly into a model that already has the same layers swapped in — build the model, `replace_linear`, then `torch.save(int8_model.state_dict(), path)`; to load, rebuild the architecture, `replace_linear` + `convert_int8_model_to_inference_mode` again, then `load_state_dict`. The image transform and tokenizer aren't affected by quantization at all — they only depend on `model_name`, so just recreate them normally with `create_model_and_transforms`/`get_tokenizer`. See the tutorial notebook for a runnable example. **Older `SwitchBackLinear`/triton path (training):** the previous `--use-bnb-linear SwitchBackLinearGlobal` / `SwitchBackLinearGlobalMemEfficient` training path, and the triton-based classes in `bitsandbytes.nn.triton_based_modules` the earlier version of this tutorial used, are effectively unusable on a fresh install today: `triton==2.0.0.post1` (which those kernels were written against) has been removed from PyPI entirely, every installable triton version reorganized `triton.language.libdevice` away, and `bitsandbytes>=0.50` dropped `triton_based_modules` outright. If you hit `AttributeError: module 'triton.language' has no attribute 'libdevice'`, that's why — it's not fixable by pinning a version anymore. Use `Linear8bitLt` (above) for inference. There's currently no equivalent replacement documented for the *training*-time SwitchBack speedup path. See the tutorial https://github.com/mlfoundations/open_clip/blob/main/tutorials/int8_tutorial.ipynb, and the original SwitchBack [paper](https://arxiv.org/abs/2304.13013) for the training-time approach (0.1pt accuracy cost vs. bf16 on CLIP ViT-Huge, per the paper — a very different regime from the post-hoc inference-only quantization measured above). ### Support for remote loading/training It is always possible to resume directly from a remote file, e.g., a file in an s3 bucket. Just set `--resume s3:// `. This will work with any filesystem supported by `fsspec`. It is also possible to train `open_clip` models while continuously backing up to s3. This can help to avoid slow local file systems. Say that your node has a local ssd `/scratch`, an s3 bucket `s3://`. In that case, set `--logs /scratch` and `--remote-sync s3://`. Then, a background process will sync `/scratch/` to `s3:///`. After syncing, the background process will sleep for `--remote-sync-frequency` seconds, which defaults to 5 minutes. There is also experimental support for syncing to other remote file systems, not just s3. To do so, specify `--remote-sync-protocol fsspec`. However, this is currently very slow and not recommended. Also, to optionally avoid saving too many checkpoints locally when using these features, you can use `--delete-previous-checkpoint` which deletes the previous checkpoint after saving a new one. Note: if you are using this feature with `--resume latest`, there are a few warnings. First, use with `--save-most-recent` is not supported. Second, only `s3` is supported. Finally, since the sync happens in the background, it is possible that the most recent checkpoint may not be finished syncing to the remote. ### Pushing Models to Hugging Face Hub The module `open_clip.push_to_hf_hub` includes helpers for pushing models /w weights and config to the HF Hub. The tool can be run from command line, ex: `python -m open_clip.push_to_hf_hub --model convnext_large_d_320 --pretrained /train/checkpoints/epoch_12.pt --repo-id laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft` ## Acknowledgments We gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this part of work by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster at Jülich Supercomputing Centre (JSC). ## The Team Current development of this repository is led by [Ross Wightman](https://rwightman.com/), [Romain Beaumont](https://github.com/rom1504), [Cade Gordon](http://cadegordon.io/), and [Vaishaal Shankar](http://vaishaal.com/). The original version of this repository is from a group of researchers at UW, Google, Stanford, Amazon, Columbia, and Berkeley. [Gabriel Ilharco*](http://gabrielilharco.com/), [Mitchell Wortsman*](https://mitchellnw.github.io/), [Nicholas Carlini](https://nicholas.carlini.com/), [Rohan Taori](https://www.rohantaori.com/), [Achal Dave](http://www.achaldave.com/), [Vaishaal Shankar](http://vaishaal.com/), [John Miller](https://people.eecs.berkeley.edu/~miller_john/), [Hongseok Namkoong](https://hsnamkoong.github.io/), [Hannaneh Hajishirzi](https://homes.cs.washington.edu/~hannaneh/), [Ali Farhadi](https://homes.cs.washington.edu/~ali/), [Ludwig Schmidt](https://people.csail.mit.edu/ludwigs/) Special thanks to [Jong Wook Kim](https://jongwook.kim/) and [Alec Radford](https://github.com/Newmu) for help with reproducing CLIP! ## Citing If you found this repository useful, please consider citing: ```bibtex @software{ilharco_gabriel_2021_5143773, author = {Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig}, title = {OpenCLIP}, month = jul, year = 2021, note = {If you use this software, please cite it as below.}, publisher = {Zenodo}, version = {0.1}, doi = {10.5281/zenodo.5143773}, url = {https://doi.org/10.5281/zenodo.5143773} } ``` ```bibtex @inproceedings{cherti2023reproducible, title={Reproducible scaling laws for contrastive language-image learning}, author={Cherti, Mehdi and Beaumont, Romain and Wightman, Ross and Wortsman, Mitchell and Ilharco, Gabriel and Gordon, Cade and Schuhmann, Christoph and Schmidt, Ludwig and Jitsev, Jenia}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={2818--2829}, year={2023} } ``` ```bibtex @inproceedings{Radford2021LearningTV, title={Learning Transferable Visual Models From Natural Language Supervision}, author={Alec Radford and Jong Wook Kim and Chris Hallacy and A. Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever}, booktitle={ICML}, year={2021} } ``` ```bibtex @inproceedings{schuhmann2022laionb, title={{LAION}-5B: An open large-scale dataset for training next generation image-text models}, author={Christoph Schuhmann and Romain Beaumont and Richard Vencu and Cade W Gordon and Ross Wightman and Mehdi Cherti and Theo Coombes and Aarush Katta and Clayton Mullis and Mitchell Wortsman and Patrick Schramowski and Srivatsa R Kundurthy and Katherine Crowson and Ludwig Schmidt and Robert Kaczmarczyk and Jenia Jitsev}, booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track}, year={2022}, url={https://openreview.net/forum?id=M3Y74vmsMcY} } ``` [](https://zenodo.org/badge/latestdoi/390536799) --- ## File: docs/clip_conceptual_captions.md ## Additional training curves for CLIP on Conceptual Captions # Zero shot accuracy # Training loss curve # Validation loss curve # Validation recall --- ## File: docs/clipa.md ## CLIPA In this work, we present a surprising finding that there exists an _inverse_ scaling law for CLIP training, whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. Moreover, we showcase that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law. As a result of this finding, we are able to successfully train CLIP even by using academic resources. For example, on an A100 eight-GPU server, our CLIP models achieve zero-shot top-1 ImageNet accuracies of **63.2%** in about **2 days**, **67.8%** in about **3 days**, and **69.3%** in about **4 days**. Moreover, We find that CLIPA at scale leads to state-of-the-art performance. For example, our CLIPA-v2 H/14 achieves a zero-shot top-1 ImageNet accuracy of **81.8%**, with a budget less than **$15000**. For more details, please see our paper [An Inverse Scaling Law for CLIP Training](https://arxiv.org/abs/2305.07017) and [CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a $10,000 Budget; An Extra $4,000 Unlocks 81.8% Accuracy](https://arxiv.org/abs/2306.15658). Eight token length reduction strategies are investigated in this work, detailed as follows. ## Image token length reduction * `resize`: use `--force-image-size` to specify the image size you want to adopt. We find this strategy generally works the best as it retains full image information. * `random mask`: Randomly mask out image patches. use `--force-patch-dropout` to specify the mask ratio you want to adopt. * `grid mask`: Preserve one patch in each 2 × 2 grid window. We do not provide implementation for grid masking, as it is only experimental and we generally find resizing works better. * `block mask`: Keep a single block and remove other patches. We do not provide implementation for block masking, as it is only experimental and we generally find resizing works better. ## Text token length reduction * `syntax mask`: Assign different masking priorities to parts of speech. Specify `"text_mask": syntax` in `"tokenizer_kwargs"` in `"text_cfg"` of model config `json` file to use. Specifically, we prioritize retaining nouns, followed by adjectives, and then other words. We find this strategy generally works the best as it retains critical information for contrastive learning. * `truncate`: Truncation selects the first N text tokens and discards the rest. This is the default setting of `open_clip`. * `random mask`: Randomly drops a portion of the text tokens. Specify `"text_mask": random` in `"tokenizer_kwargs"` in `"text_cfg"` of model config `json` file to use. * `block mask`: Randomly preserves consecutive text sequences. Specify `"text_mask": block` in `"tokenizer_kwargs"` in `"text_cfg"` of model config `json` file to use. ## Installation The installation is really the same as `open_clip`, except for the usage of Natural Language Toolkit (NLTK) in `syntax mask` of text token length reduction. Please follow the [official doc](https://www.nltk.org/) to install NLTK. Note that the the usage of NLTK brings two constraints: * Because certain functions like `nltk.pos_tag` from NLTK only support English and Russian for now, the `syntax mask` only works for English. we have not tested it on Russian or any other language. Theoretically, it should work the same, given a proper language processing toolkit for other languages. If you still want to apply `syntax mask` on other languages, try finding the right toolkit. Otherwise, use other text token length reduction strategies * some modules of NLTK like `punkt` or `averaged_perceptron_tagger` need to be downloaded first before using NLTK. We have included the downloading code in `tokenizer.py`, but this might cause trouble in certain cases. You may want to manually download those modules first, by `nltk.download('punkt')` and `nltk.download('averaged_perceptron_tagger')`, and then setup the environmental variable before running the script `export NLTK_DATA=cache`. Note that this is a one-time effort. Remember to comment out those `nltk.download` lines in `tokenizer.py` afterwards. ## Training We provide example scripts to reproduce our CLIPA results on an A100 eight-GPU machine under path `docs/script_examples/clipa`. For instance, to reproduce the CLIPA-L16(I37,T8) results, first run the pre-training script ``` bash docs/script_examples/clipa/vit_l16/i37_t8_pretrain.sh ``` and fine-tune the pre-trained checkpoint with ``` bash docs/script_examples/clipa/vit_l16/i37_t8_finetune.sh ``` - Remember to change the path to dataset to your own path. - This is a two-stage training pipeline. Remember to change the path to pre-trained checkpoint to your own when fine-tuning. - The training time is ~3 days for pre-training and ~1 day for fine-tuning on an A100 eight-GPU machine. ## Model Weights Below are CLIPA trained weights on LAION-400M with an A100 eight-GPU machine. All models are pre-trained for 6 epochs with reduced input token lengths and subsequently fine-tuned for 0.36 epoch with full input token lengths. | | Pre-trained Weights | zero-shot IN-1K | |---------------------|:----------------------------------------------------------------------------------------------:|:---------------:| | CLIPA-B/16(I50,T16) | [download](https://drive.google.com/file/d/1MDpz8gV2Vjaazk16rBhLxU8811U7_cGL/view?usp=sharing) | 59.7 | | CLIPA-L/16(I17,T16) | [download](https://drive.google.com/file/d/1Tr2GYiKAaMH6EGIn5l7eX_1K20eaA3WA/view?usp=sharing) | 60.3 | | CLIPA_L/16(I37,T8) | [download](https://drive.google.com/file/d/1EM1ChRNARpLckkJjf6m7njCY3xyvpGBu/view?usp=sharing) | 57.9 | | | Fine-tuned Weights | zero-shot IN-1K | |---------------------|:----------------------------------------------------------------------------------------------:|:-----:| | CLIPA-B/16(I50,T16) | [download](https://drive.google.com/file/d/1fURK0K_a3-83jVEI4PVEbnEJb_V6UbGv/view?usp=sharing) | 63.2 | | CLIPA-L/16(I17,T16) | [download](https://drive.google.com/file/d/18qqZGOTGOgb3I3JWONuat6qObsgLq7sR/view?usp=sharing) | 67.8 | | CLIPA_L/16(I37,T8) | [download](https://drive.google.com/file/d/1lV7pLORUK04T9QKKx9TpYtMws-AZrib0/view?usp=sharing) | 69.3 | ## CLIPA-v2 We also provide example scripts to reproduce our CLIPA-v2 H/14 results under path `docs/script_examples/clipav2`. Note that the original results are obtained with [our JAX implementation](https://github.com/UCSC-VLAA/CLIPA/tree/master/clipa_jax). These scripts are written after manually scanning the JAX config files. As it is infeasible for us to retrain those models again with pytorch, its correctness cannot be verified with 100% confidence. Use them at your own discretion. --- ## File: docs/datacomp_models.md ## CommonPool and DataComp models As part of [DataComp](https://github.com/mlfoundations/datacomp), we trained models on CommonPool using various data filtering strategies. We release models for all four scales of the competition, small, medium, large and xlarge, corresponding to a pool size and number of samples seen of 12.8M, 128M, 1.28B and 12.8B, respectively. The models are specified below, see our paper [DataComp: In seearch of the next generation of multimodal datasets](https://arxiv.org/abs/2304.14108) for more details. ## xlarge scale models * `datacomp_xl_s13b_b90k`: A ViT-L/14 trained on DataComp-1B for 12.8B steps and batch size 90k. Achieves 79.2% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K. * `commonpool_xl_clip_s13b_b90k`: A ViT-L/14 trained on CommonPool-XL filtered using CLIP scores, for 12.8B steps and batch size 90k. Achieves 76.4% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-CommonPool.XL.clip-s13B-b90K. * `commonpool_xl_laion_s13b_b90k`: A ViT-L/14 trained on CommonPool-XL filtered using the LAION-2B filtering scheme, for 12.8B steps and batch size 90k. Achieves 75.5% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-CommonPool.XL.laion-s13B-b90K. * `commonpool_xl_s13b_b90k`: A ViT-L/14 trained on CommonPool-XL without any filtering, for 12.8B steps and batch size 90k. Achieves 72.3% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-CommonPool.XL-s13B-b90K. ## large scale models * `datacomp_l_s1b_b8k`: A ViT-B/16 trained on a 140M subset of DataComp-1B, for 1.28B steps and batch size 8k. Achieves 63.1% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-16-DataComp.L-s1B-b8K. * `commonpool_l_clip_s1b_b8k`: A ViT-B/16 trained on CommonPool-L filtered using CLIP scores, for 1.28B steps and batch size 8k. Achieves 57.8% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-16-CommonPool.L.clip-s1B-b8K. * `commonpool_l_laion_s1b_b8k`: A ViT-B/16 trained on CommonPool-L filtered using the LAION-2B filtering scheme, for 1.28B steps and batch size 8k. Achieves 55.3% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-16-CommonPool.L.laion-s1B-b8K. * `commonpool_l_image_s1b_b8k`: A ViT-B/16 trained on CommonPool-L filtered using image-based filtering, for 1.28B steps and batch size 8k. Achieves 57.2% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-16-CommonPool.L.image-s1B-b8K. * `commonpool_l_text_s1b_b8k`: A ViT-B/16 trained on CommonPool-L filtered using text-based filtering, for 1.28B steps and batch size 8k. Achieves 56.1% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-16-CommonPool.L.text-s1B-b8K. * `commonpool_l_basic_s1b_b8k`: A ViT-B/16 trained on CommonPool-L filtered using basic filtering (English filtering + caption length and image size filtering), for 1.28B steps and batch size 8k. Achieves 51.6% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-16-CommonPool.L.basic-s1B-b8K. * `commonpool_l_s1b_b8k`: A ViT-B/16 trained on CommonPool-L without any filtering, for 1.28B steps and batch size 8k. Achieves 45.9% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-16-CommonPool.L-s1B-b8K. ## medium scale models * `datacomp_m_s128m_b4k`: A ViT-B/32 trained on a 14M subset of DataComp-1B, for 128M steps and batch size 4k. Achieves 29.7% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-DataComp.M-s128M-b4K. * `commonpool_m_clip_s128m_b4k`: A ViT-B/32 trained on CommonPool-M filtered using CLIP scores, for 128M steps and batch size 4k. Achieves 27.3% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.M.clip-s128M-b4K. * `commonpool_m_laion_s128m_b4k`: A ViT-B/32 trained on CommonPool-M filtered using the LAION-2B filtering scheme, for 128M steps and batch size 4k. Achieves 23.0% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.M.laion-s128M-b4K. * `commonpool_m_image_s128m_b4k`: A ViT-B/32 trained on CommonPool-M filtered using image-based filtering, for 128M steps and batch size 4k. Achieves 26.8% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.M.image-s128M-b4K. * `commonpool_m_text_s128m_b4k`: A ViT-B/32 trained on CommonPool-M filtered using text-based filtering, for 128M steps and batch size 4k. Achieves 25.5% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.M.text-s128M-b4K. * `commonpool_m_basic_s128m_b4k`: A ViT-B/32 trained on CommonPool-M filtered using basic filtering (English filtering + caption length and image size filtering), for 128M steps and batch size 4k. Achieves 22.6% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.M.basic-s128M-b4K. * `commonpool_m_s128m_b4k`: A ViT-B/32 trained on CommonPool-M without any filtering, for 128M steps and batch size 4k. Achieves 17.6% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.M-s128M-b4K. ## small scale models * `datacomp_s_s13m_b4k`: A ViT-B/32 trained on a 1.4M subset of DataComp-1B, for 12.8M steps and batch size 4k. Achieves 3.9% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-DataComp.S-s13M-b4K. * `commonpool_s_clip_s13m_b4k`: A ViT-B/32 trained on CommonPool-S filtered using CLIP scores, for 12.8M steps and batch size 4k. Achieves 5.1% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.S.clip-s13M-b4K. * `commonpool_s_laion_s13m_b4k`: A ViT-B/32 trained on CommonPool-S filtered using the LAION-2B filtering scheme scores, for 12.8M steps and batch size 4k. Achieves 3.1% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.S.laion-s13M-b4K. * `commonpool_s_image_s13m_b4k`: A ViT-B/32 trained on CommonPool-S filtered using image-based filtering, for 12.8M steps and batch size 4k. Achieves 4.3% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.S.image-s13M-b4K. * `commonpool_s_text_s13m_b4k`: A ViT-B/32 trained on CommonPool-S filtered using text-based filtering, for 12.8M steps and batch size 4k. Achieves 4.6% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.S.text-s13M-b4K. * `commonpool_s_basic_s13m_b4k`: A ViT-B/32 trained on CommonPool-S filtered using basic filtering (English filtering + caption length and image size filtering), for 12.8M steps and batch size 4k. Achieves 3.0% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.S.basic-s13M-b4K. * `commonpool_s_s13m_b4k`: A ViT-B/32 trained on CommonPool-S without any filtering, for 12.8M steps and batch size 4k. Achieves 2.5% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-B-32-CommonPool.S-s13M-b4K. --- ## File: docs/LOW_ACC.md As we describe in more detail below, CLIP models in a medium accuracy regime already allow us to draw conclusions about the robustness of larger CLIP models since the models follow reliable scaling laws. [Cherti et al., 2022](https://arxiv.org/abs/2212.07143) and [Gadre et al., 2023](https://arxiv.org/abs/2304.14108) show additional discussions about the scaling behavior of CLIP models. ## Scaling trends The plot below shows how zero-shot performance of CLIP models varies as we scale the number of samples used for training. Zero-shot performance increases steadily for both ImageNet and [ImageNetV2](https://arxiv.org/abs/1902.10811), and is far from saturated at ~15M samples. ## Why are low-accuracy CLIP models interesting? **TL;DR:** CLIP models have high effective robustness, even at small scales. CLIP models are particularly intriguing because they are more robust to natural distribution shifts (see Section 3.3 in the [CLIP paper](https://arxiv.org/abs/2103.00020)). This phenomena is illustrated by the figure below, with ImageNet accuracy on the x-axis and [ImageNetV2](https://arxiv.org/abs/1902.10811) (a reproduction of the ImageNet validation set with distribution shift) accuracy on the y-axis. Standard training denotes training on the ImageNet train set and the CLIP zero-shot models are shown as stars. As observed by [Taori et al., 2020](https://arxiv.org/abs/2007.00644) and [Miller et al., 2021](https://arxiv.org/abs/2107.04649), the in-distribution and out-of-distribution accuracies of models trained on ImageNet follow a predictable linear trend (the red line in the above plot). *Effective robustness* quantifies robustness as accuracy beyond this baseline, i.e., how far a model lies above the red line. Ideally a model would not suffer from distribution shift and fall on the y = x line ([trained human labelers are within a percentage point of the y = x line](http://proceedings.mlr.press/v119/shankar20c.html)). Even though the CLIP models trained with this codebase achieve much lower accuracy than those trained by OpenAI, our models still lie on the same trend of improved effective robustness (the purple line). Therefore, we can study what makes CLIP robust without requiring industrial-scale compute. For more information on effective robustness, please see: - [Recht et al., 2019](https://arxiv.org/abs/1902.10811). - [Taori et al., 2020](https://arxiv.org/abs/2007.00644). - [Miller et al., 2021](https://arxiv.org/abs/2107.04649). To know more about the factors that contribute to CLIP's robustness refer to [Fang et al., 2022](https://arxiv.org/abs/2205.01397). --- ## File: docs/PRETRAINED.md ## Pretrained model results We evaluate the full collection of available models on a suite of 38 datasets in a zero-shot setting (i.e., without fine-tuning), following [Gadre et al., 2023](https://arxiv.org/abs/2304.14108). Click below to see the full results. - [Full results (English)](openclip_results.csv) - [Classification-only results](openclip_classification_results.csv) - [Retrieval results](openclip_retrieval_results.csv) - [Multilingual retrieval results](openclip_multilingual_retrieval_results.csv) ## Pretrained model details Below are details for several of our pretrained models. ### LAION-400M - https://laion.ai/laion-400-open-dataset We ran experiments in an attempt to reproduce OpenAI's ViT results with the comparably sized (and open) LAION-400M dataset. Trained weights can be found in release [v0.2](https://github.com/mlfoundations/open_clip/releases/tag/v0.2-weights). The LAION400M weights have been trained on the JUWELS supercomputer (see acknowledgements section below). #### ViT-B/32 224x224 We replicate OpenAI's results on ViT-B/32, reaching a top-1 ImageNet-1k zero-shot accuracy of 62.96%. **Zero-shot comparison (courtesy of Andreas Fürst)** ViT-B/32 was trained with 128 A100 (40 GB) GPUs for ~36 hours, 4600 GPU-hours. The per-GPU batch size was 256 for a global batch size of 32768. 256 is much lower than it could have been (~320-384) due to being sized initially before moving to 'local' contrastive loss. #### ViT-B/16 224x224 The B/16 LAION400M training reached a top-1 ImageNet-1k zero-shot validation score of 67.07. This was the first major train session using the updated webdataset 0.2.x code. A bug was found that prevented shards from being shuffled properly between nodes/workers each epoch. This was fixed part way through training (epoch 26) but likely had an impact. ViT-B/16 was trained with 176 A100 (40 GB) GPUS for ~61 hours, 10700 GPU-hours. Batch size per GPU was 192 for a global batch size of 33792. #### ViT-B/16+ 240x240 The B/16+ 240x240 LAION400M training reached a top-1 ImageNet-1k zero-shot validation score of 69.21. This model is the same depth as the B/16, but increases the - vision width from 768 -> 896 - text width from 512 -> 640 - the resolution 224x224 -> 240x240 (196 -> 225 tokens) Unlike the B/16 run above, this model was a clean run with no dataset shuffling issues. ViT-B/16+ was trained with 224 A100 (40 GB) GPUS for ~61 hours, 13620 GPU-hours. Batch size per GPU was 160 for a global batch size of 35840. #### ViT-L/14 224x224 The L/14 LAION-400M training reached a top-1 ImageNet-1k zero-shot validation score of 72.77. ViT-L/14 was trained with 400 A100 (40 GB) GPUS for ~127 hours, 50800 GPU-hours. Batch size per GPU was 96 for a global batch size of 38400. Grad checkpointing was enabled. ### LAION-2B (en) - https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/ A ~2B sample subset of LAION-5B with english captions (https://huggingface.co/datasets/laion/laion2B-en) #### ViT-B/32 224x224 A ViT-B/32 trained on LAION-2B, reaching a top-1 ImageNet-1k zero-shot accuracy of 65.62%. ViT-B/32 was trained with 112 A100 (40 GB) GPUs. The per-GPU batch size was 416 for a global batch size of 46592. Compute generously provided by [stability.ai](https://stability.ai/). A second iteration of B/32 was trained on stability.ai cluster with a larger global batch size and learning rate, hitting 66.6% top-1. See https://huggingface.co/laion/CLIP-ViT-B-32-laion2B-s34B-b79K #### ViT-L/14 224x224 A ViT-L/14 with a 75.3% top-1 ImageNet-1k zero-shot was trained on JUWELS Booster. See model details here https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K These weights use a different dataset mean and std than others. Instead of using the OpenAI mean & std, inception style normalization `[-1, 1]` is used via a mean and std of `[0.5, 0.5, 0.5]`. This is handled automatically if using `open_clip.create_model_and_transforms` from pretrained weights. #### ViT-H/14 224x224 A ViT-H/14 with a 78.0% top-1 ImageNet-1k zero-shot was trained on JUWELS Booster. See model details here https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K #### ViT-g/14 224x224 A ViT-g/14 with a 76.6% top-1 ImageNet-1k zero-shot was trained on JUWELS Booster. See model details here https://huggingface.co/laion/CLIP-ViT-g-14-laion2B-s12B-b42K This model was trained with a shorted schedule than other LAION-2B models with 12B samples seen instead of 32+B. It matches LAION-400M training in samples seen. Many zero-shot results are lower as a result, but despite this it performs very well in some OOD zero-shot and retrieval tasks. #### ViT-B/32 roberta base A ViT-B/32 with roberta base encoder with a 61.7% top-1 ImageNet-1k zero-shot was trained on stability. See model details here https://huggingface.co/laion/CLIP-ViT-B-32-roberta-base-laion2B-s12B-b32k This is the first openclip model using a HF text tower. It has better performance on a range of tasks compared to the standard text encoder, see [metrics](https://huggingface.co/laion/CLIP-ViT-B-32-roberta-base-laion2B-s12B-b32k/blob/main/unknown.png) #### ViT-B/32 xlm roberta base A ViT-B/32 with xlm roberta base encoder with a 62.33% top-1 ImageNet-1k zero-shot was trained on stability. See model details here https://huggingface.co/laion/CLIP-ViT-B-32-xlm-roberta-base-laion5B-s13B-b90k This is the first openclip model trained on the full laion5B dataset; hence the first multilingual clip trained with openclip. It has better performance on a range of tasks compared to the standard text encoder, see [metrics](https://huggingface.co/laion/CLIP-ViT-B-32-xlm-roberta-base-laion5B-s13B-b90k/blob/main/metrics.png) A preliminary multilingual evaluation was run: 43% on imagenet1k italian (vs 21% for english B/32), 37% for imagenet1k japanese (vs 1% for english B/32 and 50% for B/16 clip japanese). It shows the multilingual property is indeed there as expected. Larger models will get even better performance. #### ViT-H/14 xlm roberta large A ViT-H/14 with xlm roberta large encoder with a 77.0% (vs 78% for the english equivalent) top-1 ImageNet-1k zero-shot was trained on stability. See model details here https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k This model was trained following the [LiT](https://arxiv.org/abs/2111.07991) methodology: the image tower was frozen (initialized from english openclip ViT-H/14), the text tower was initialized from [xlm roberta large](https://huggingface.co/xlm-roberta-large) and unfrozen. This reduced training cost by a 3x factor. See full english [metrics](https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k/resolve/main/results_xlm_roberta_large.png) On zero shot classification on imagenet with translated prompts this model reaches: - 56% in italian (vs 21% for https://github.com/clip-italian/clip-italian) - 53% in japanese (vs 54.6% for https://github.com/rinnakk/japanese-clip) - 55.7% in chinese (to be compared with https://github.com/OFA-Sys/Chinese-CLIP) #### YFCC-15M Below are checkpoints of models trained on YFCC-15M, along with their zero-shot top-1 accuracies on ImageNet and ImageNetV2. These models were trained using 8 GPUs and the same hyperparameters described in the "Sample running code" section, with the exception of `lr=5e-4` and `epochs=32`. - [ResNet-50](https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/rn50-quickgelu-yfcc15m-455df137.pt) (32.7% / 27.9%) - [ResNet-101](https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/rn101-quickgelu-yfcc15m-3e04b30e.pt) (34.8% / 30.0%) #### CC12M - https://github.com/google-research-datasets/conceptual-12m - [ResNet-50](https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/rn50-quickgelu-cc12m-f000538c.pt) (36.45%) ### CommonPool and DataComp models As part of [DataComp](https://github.com/mlfoundations/datacomp), we trained models on CommonPool using various data filtering strategies. The best performing models are specified below for the xlarge scale, see our paper [DataComp: In seearch of the next generation of multimodal datasets](https://arxiv.org/abs/2304.14108) for more details. Additional models and more information can be found at [/docs/datacomp_models.md](/docs/datacomp_models.md). - `datacomp_xl_s13b_b90k`: A ViT-L/14 trained on DataComp-1B for 12.8B steps and batch size 90k. Achieves 79.2% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K. - `commonpool_xl_clip_s13b_b90k`: A ViT-L/14 trained on CommonPool-XL filtered using CLIP scores, for 12.8B steps and batch size 90k. Achieves 76.4% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-CommonPool.XL.clip-s13B-b90K. - `commonpool_xl_laion_s13b_b90k`: A ViT-L/14 trained on CommonPool-XL filtered using the LAION-2B filtering scheme, for 12.8B steps and batch size 90k. Achieves 75.5% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-CommonPool.XL.laion-s13B-b90K. - `commonpool_xl_s13b_b90k`: A ViT-L/14 trained on CommonPool-XL without any filtering, for 12.8B steps and batch size 90k. Achieves 72.3% zero-shot accuracy on ImageNet. Available at https://huggingface.co/laion/CLIP-ViT-L-14-CommonPool.XL-s13B-b90K. If you use models trained on DataComp-1B or CommonPool variations, please consider citing the following: ```bibtex @article{datacomp, title={DataComp: In search of the next generation of multimodal datasets}, author={Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt}, journal={arXiv preprint arXiv:2304.14108}, year={2023} } ``` ### MetaCLIP MetaCLIP models are described in the paper [Demystifying CLIP Data](https://arxiv.org/abs/2309.16671). These models were developed by Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer and Christoph Feichtenhofer from Meta, New York University and the University of Washington. Models are licensed under CC-BY-NC. More details are available at https://github.com/facebookresearch/MetaCLIP. If you use MetaCLIP models, please cite the following: ```bibtex @inproceedings{xu2023metaclip, title={Demystifying CLIP Data}, author={Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu, Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer and Christoph Feichtenhofer}, journal={arXiv preprint arXiv:2309.16671}, year={2023} } ``` ### EVA-CLIP EVA-CLIP models are described in the paper [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://arxiv.org/abs/2303.15389). These models were developed by Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang and Yue Cao from BAAI and HUST. Models are licensed under the MIT License. More details are available at https://github.com/baaivision/EVA/tree/master/EVA-CLIP. If you use EVA models, please cite the following: ```bibtex @article{EVA-CLIP, title={EVA-CLIP: Improved Training Techniques for CLIP at Scale}, author={Sun, Quan and Fang, Yuxin and Wu, Ledell and Wang, Xinlong and Cao, Yue}, journal={arXiv preprint arXiv:2303.15389}, year={2023} } ``` ### NLLB-CLIP NLLB-CLIP models are described in the paper [NLLB-CLIP - train performant multilingual image retrieval model on a budget](https://arxiv.org/abs/2309.01859) by Alexander Visheratin. The model was trained following the [LiT](https://arxiv.org/abs/2111.07991) methodology: the image tower was frozen, the text tower was initialized from the [NLLB](https://arxiv.org/abs/2207.04672) encoder and unfrozen. The model was trained on the [LAION-COCO-NLLB](https://huggingface.co/datasets/visheratin/laion-coco-nllb) dataset. The first version of the model (`nllb-clip`) described in the paper was trained using the OpenAI CLIP image encoder. The second version of the model (`nllb-clip-siglip`) was trained using the [SigLIP](https://arxiv.org/abs/2303.15343) image encoder. Models are licensed under CC-BY-NC. If you use NLLB-CLIP models, please cite the following: ```bibtex @article{visheratin2023nllb, title={NLLB-CLIP--train performant multilingual image retrieval model on a budget}, author={Visheratin, Alexander}, journal={arXiv preprint arXiv:2309.01859}, year={2023} } ``` ### CLIPA CLIPA models are described in the following papers by Xianhang Li, Zeyu Wang, Cihang Xie from UC Santa Cruz: - [An Inverse Scaling Law for CLIP Training](https://arxiv.org/abs/2305.07017) - [CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a $10,000 Budget; An Extra $4,000 Unlocks 81.8% Accuracy](https://arxiv.org/abs/2306.15658) Models are licensed under Apache 2.0. More details are available at https://github.com/UCSC-VLAA/CLIPA and [here](clipa.md). If you use CLIPA models, please cite the following: ```bibtex @inproceedings{li2023clipa, title={An Inverse Scaling Law for CLIP Training}, author={Xianhang Li and Zeyu Wang and Cihang Xie}, booktitle={NeurIPS}, year={2023}, } ``` ```bibtex @article{li2023clipav2, title={CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a $10,000 Budget; An Extra $4,000 Unlocks 81.8% Accuracy}, author={Xianhang Li and Zeyu Wang and Cihang Xie}, journal={arXiv preprint arXiv:2306.15658}, year={2023}, } ``` ### SigLIP SigLIP models are described in the paper [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343). These models were developed by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer from Google Deepmind. Models are licensed under the Apache 2 license. More details are available at hhttps://github.com/google-research/big_vision. If you use SigLIP models, please cite the following: ```bibtex @article{zhai2023sigmoid, title={Sigmoid loss for language image pre-training}, author={Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas}, journal={arXiv preprint arXiv:2303.15343}, year={2023} } ``` ### DFN Data Filtering Network models are described in https://arxiv.org/abs/2309.17425. These models were developed by Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev and Vaishaal Shankar from Apple and the University of Washington. Models are licensed under the following: https://huggingface.co/apple/DFN5B-CLIP-ViT-H-14-384/blob/main/LICENSE. If you use DFN models, please cite the following: ```bibtext @article{fang2023data, title={Data Filtering Networks}, author={Fang, Alex and Jose, Albin Madappally and Jain, Amit and Schmidt, Ludwig and Toshev, Alexander and Shankar, Vaishaal}, journal={arXiv preprint arXiv:2309.17425}, year={2023} } ```