# Technical Documentation: huggingface/transformers > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/huggingface/transformers) ### Overview (`README.md`)

Hugging Face Transformers Library

Checkpoints on Hub Build GitHub Documentation GitHub release Contributor Covenant DOI

English | 简体中文 | 繁體中文 | 한국어 | Español | 日本語 | हिन्दी | मराठी | Русский | Português | తెలుగు | Français | Deutsch | Italiano | Tiếng Việt | العربية | اردو | বাংলা | فارسی | Română | Türkçe

State-of-the-art pretrained models for inference and training

Transformers acts as the model-definition framework for state-of-the-art machine learning with text, computer vision, audio, video, and multimodal models, for both inference and training. It centralizes the model definition so that this definition is agreed upon across the ecosystem. `transformers` is the pivot across frameworks: if a model definition is supported, it will be compatible with the majority of training frameworks (Axolotl, Unsloth, DeepSpeed, FSDP, PyTorch-Lightning, ...), inference engines (vLLM, SGLang, TGI, ...), and adjacent modeling libraries (llama.cpp, mlx, ...) which leverage the model definition from `transformers`. We pledge to help support new state-of-the-art models and democratize their usage by having their model definition be simple, customizable, and efficient. There are over 1M+ Transformers [model checkpoints](https://huggingface.co/models?library=transformers&sort=trending) on the [Hugging Face Hub](https://huggingface.co/models) you can use. Explore the [Hub](https://huggingface.co/) today to find a model and use Transformers to help you get started right away. ## Installation Transformers works with Python 3.11+, and [PyTorch](https://pytorch.org/get-started/locally/) 2.6+. Create and activate a virtual environment with [venv](https://docs.python.org/3/library/venv.html) or [uv](https://docs.astral.sh/uv/), a fast Rust-based Python package and project manager. ```py # venv python -m venv .my-env source .my-env/bin/activate # uv uv venv .my-env source .my-env/bin/activate ``` Install Transformers in your virtual environment. ```py # pip pip install "transformers[torch]" # uv uv pip install "transformers[torch]" ``` Install Transformers from source if you want the latest changes in the library or are interested in contributing. However, the *latest* version may not be stable. Feel free to open an [issue](https://github.com/huggingface/transformers/issues) if you encounter an error. ```shell git clone https://github.com/huggingface/transformers.git cd transformers # pip pip install '.[torch]' # uv uv pip install '.[torch]' ``` ## Quickstart Get started with Transformers right away with the [Pipeline](https://huggingface.co/docs/transformers/pipeline_tutorial) API. The `Pipeline` is a high-level inference class that supports text, audio, vision, and multimodal tasks. It handles preprocessing the input and returns the appropriate output. Instantiate a pipeline and specify model to use for text generation. The model is downloaded and cached so you can easily reuse it again. Finally, pass some text to prompt the model. ```py from transformers import pipeline pipeline = pipeline(task="text-generation", model="Qwen/Qwen2.5-1.5B") pipeline("the secret to baking a really good cake is ") [{'generated_text': 'the secret to baking a really good cake is 1) to use the right ingredients and 2) to follow the recipe exactly. the recipe for the cake is as follows: 1 cup of sugar, 1 cup of flour, 1 cup of milk, 1 cup of butter, 1 cup of eggs, 1 cup of chocolate chips. if you want to make 2 cakes, how much sugar do you need? To make 2 cakes, you will need 2 cups of sugar.'}] ``` To chat with a model, the usage pattern is the same. The only difference is you need to construct a chat history (the input to `Pipeline`) between you and the system. > [!TIP] > You can also chat with a model directly from the command line, as long as [`transformers serve` is running](https://huggingface.co/docs/transformers/main/en/serving). > ```shell > transformers chat Qwen/Qwen2.5-0.5B-Instruct > ``` ```py import torch from transformers import pipeline chat = [ {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."}, {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"} ] pipeline = pipeline(task="text-generation", model="meta-llama/Meta-Llama-3-8B-Instruct", dtype=torch.bfloat16, device_map="auto") response = pipeline(chat, max_new_tokens=512) print(response[0]["generated_text"][-1]["content"]) ``` Expand the examples below to see how `Pipeline` works for different modalities and tasks.
Automatic speech recognition ```py from transformers import pipeline pipeline = pipeline(task="automatic-speech-recognition", model="openai/whisper-large-v3") pipeline("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac") {'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'} ```
Image classification

```py from transformers import pipeline pipeline = pipeline(task="image-classification", model="facebook/dinov2-small-imagenet1k-1-layer") pipeline("https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png") [{'label': 'macaw', 'score': 0.997848391532898}, {'label': 'sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita', 'score': 0.0016551691805943847}, {'label': 'lorikeet', 'score': 0.00018523589824326336}, {'label': 'African grey, African gray, Psittacus erithacus', 'score': 7.85409429227002e-05}, {'label': 'quail', 'score': 5.502637941390276e-05}] ```
Visual question answering

```py from transformers import pipeline pipeline = pipeline(task="visual-question-answering", model="Salesforce/blip-vqa-base") pipeline( image="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg", question="What is in the image?", ) [{'answer': 'statue of liberty'}] ```
## Why should I use Transformers? 1. Easy-to-use state-of-the-art models: - High performance on natural language understanding & generation, computer vision, audio, video, and multimodal tasks. - Low barrier to entry for researchers, engineers, and developers. - Few user-facing abstractions with just three classes to learn. - A unified API for using all our pretrained models. 1. Lower compute costs, smaller carbon footprint: - Share trained models instead of training from scratch. - Reduce compute time and production costs. - Hundreds of model architectures with 1M+ pretrained checkpoints across all modalities. 1. Choose the right framework for every part of a model's lifetime: - Train state-of-the-art models in 3 lines of code. - Move a single model between PyTorch/JAX/TF2.0 frameworks at will. - Pick the right framework for training, evaluation, and production. 1. Easily customize a model or an example to your needs: - We provide examples for each architecture to reproduce the results published by its original authors. - Model internals are exposed as consistently as possible. - Model files can be used independently of the library for quick experiments. Hugging Face Enterprise Hub
## When shouldn't I use Transformers? - This library is not a modular toolbox of building blocks for neural nets. The code in the model files is not refactored with additional abstractions on purpose, so that researchers can quickly iterate on each of the models without diving into additional abstractions/files. - The training API is optimized to work with PyTorch models provided by Transformers. For generic machine learning loops, you should use another library like [Accelerate](https://huggingface.co/docs/accelerate). ## 100 projects using Transformers Transformers is more than a toolkit to use pretrained models, it's a community of projects built around it and the Hugging Face Hub. We want Transformers to enable developers, researchers, students, professors, engineers, and anyone else to build their dream projects. In order to celebrate Transformers 100,000 stars, we wanted to put the spotlight on the community with the [awesome-transformers](./awesome-transformers.md) page which lists 100 incredible projects built with Transformers. If you own or use a project that you believe should be part of the list, please open a PR to add it! ## Example models You can test most of our models directly on their [Hub model pages](https://huggingface.co/models). Expand each modality below to see a few example models for various use cases.
Audio - Audio classification with [CLAP](https://huggingface.co/laion/clap-htsat-fused) - Automatic speech recognition with [Parakeet](https://huggingface.co/nvidia/parakeet-ctc-1.1b#transcribing-using-transformers-%F0%9F%A4%97), [Whisper](https://huggingface.co/openai/whisper-large-v3-turbo), [GLM-ASR](https://huggingface.co/zai-org/GLM-ASR-Nano-2512) and [Moonshine-Streaming](https://huggingface.co/UsefulSensors/moonshine-streaming-medium) - Keyword spotting with [Wav2Vec2](https://huggingface.co/superb/wav2vec2-base-superb-ks) - Speech to speech generation with [Moshi](https://huggingface.co/kyutai/moshiko-pytorch-bf16) - Text to audio with [MusicGen](https://huggingface.co/facebook/musicgen-large) - Text to speech with [CSM](https://huggingface.co/sesame/csm-1b)
Computer vision - Automatic mask generation with [SAM](https://huggingface.co/facebook/sam-vit-base) - Depth estimation with [DepthPro](https://huggingface.co/apple/DepthPro-hf) - Image classification with [DINO v2](https://huggingface.co/facebook/dinov2-base) - Keypoint detection with [SuperPoint](https://huggingface.co/magic-leap-community/superpoint) - Keypoint matching with [SuperGlue](https://huggingface.co/magic-leap-community/superglue_outdoor) - Object detection with [RT-DETRv2](https://huggingface.co/PekingU/rtdetr_v2_r50vd) - Pose Estimation with [VitPose](https://huggingface.co/usyd-community/vitpose-base-simple) - Universal segmentation with [OneFormer](https://huggingface.co/shi-labs/oneformer_ade20k_swin_large) - Video classification with [VideoMAE](https://huggingface.co/MCG-NJU/videomae-large)
Multimodal - Audio or text to text with [Voxtral](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507), [Audio Flamingo](https://huggingface.co/nvidia/audio-flamingo-3-hf) - Document question answering with [LayoutLMv3](https://huggingface.co/microsoft/layoutlmv3-base) - Image or text to text with [Qwen-VL](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) - Image captioning [BLIP-2](https://huggingface.co/Salesforce/blip2-opt-2.7b) - OCR-based document understanding with [GOT-OCR2](https://huggingface.co/stepfun-ai/GOT-OCR-2.0-hf) - Table question answering with [TAPAS](https://huggingface.co/google/tapas-base) - Unified multimodal understanding and generation with [Emu3](https://huggingface.co/BAAI/Emu3-Gen) - Vision to text with [Llava-OneVision](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf) - Visual question answering with [Llava](https://huggingface.co/llava-hf/llava-1.5-7b-hf) - Visual referring expression segmentation with [Kosmos-2](https://huggingface.co/microsoft/kosmos-2-patch14-224)
NLP - Masked word completion with [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) - Named entity recognition with [Gemma](https://huggingface.co/google/gemma-2-2b) - Question answering with [Mixtral](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) - Summarization with [BART](https://huggingface.co/facebook/bart-large-cnn) - Translation with [T5](https://huggingface.co/google-t5/t5-base) - Text generation with [Llama](https://huggingface.co/meta-llama/Llama-3.2-1B) - Text classification with [Qwen](https://huggingface.co/Qwen/Qwen2.5-0.5B)
## Citation We now have a [paper](https://aclanthology.org/2020.emnlp-demos.6/) you can cite for the 🤗 Transformers library: ```bibtex @inproceedings{wolf-etal-2020-transformers, title = "Transformers: State-of-the-Art Natural Language Processing", author = "Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush", booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations", month = oct, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2020.emnlp-demos.6/", pages = "38--45" } ``` --- ### Overview (`benchmark/README.md`) # Benchmarks You might want to add new benchmarks. You will need to define a python function named `run_benchmark` in your python file and the file must be located in this `benchmark/` directory. The expected function signature is the following: ```py def run_benchmark(logger: Logger, branch: str, commit_id: str, commit_msg: str, num_tokens_to_generate=100): ``` ## Writing metrics to the database `MetricsRecorder` is thread-safe, in the sense of the python [`Thread`](https://docs.python.org/3/library/threading.html#threading.Thread). This means you can start a background thread to do the readings on the device measurements while not blocking the main thread to execute the model measurements. cf [`llama.py`](./llama.py) to see an example of this in practice. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Overview (`benchmark_v2/README.md`) # Benchmarking v2 A comprehensive benchmarking framework for transformer models that supports multiple execution modes (eager, compiled, kernelized), detailed performance metrics collection, and structured output format. ## Quick Start ### Running All Benchmarks ```bash # Run all benchmarks with default settings python run_benchmarks.py # Specify output directory python run_benchmarks.py --output-dir my_results # Run with custom parameters python run_benchmarks.py \ --warmup-iterations 5 \ --measurement-iterations 10 \ --num-tokens-to-generate 200 ``` ### Uploading Results to HuggingFace Dataset You can automatically upload benchmark results to a HuggingFace Dataset for tracking and analysis: ```bash # Upload to a public dataset with auto-generated run ID python run_benchmarks.py --upload-to-hub username/benchmark-results # Upload with a custom run ID for easy identification python run_benchmarks.py --upload-to-hub username/benchmark-results --run-id experiment_v1 # Upload with custom HuggingFace token (if not set in environment) python run_benchmarks.py --upload-to-hub username/benchmark-results --token hf_your_token_here ``` **Dataset Directory Structure:** ``` dataset_name/ ├── 2025-01-15/ │ ├── runs/ # Non-scheduled runs (manual, PR, etc.) │ │ └── 123-1245151651/ # GitHub run number and ID │ │ └── benchmark_results/ │ │ ├── benchmark_summary_20250115_143022.json │ │ └── model-name/ │ │ └── model-name_benchmark_20250115_143022.json │ └── benchmark_results_abc123de/ # Scheduled runs (daily CI) │ ├── benchmark_summary_20250115_143022.json │ └── model-name/ │ └── model-name_benchmark_20250115_143022.json └── 2025-01-16/ └── ... ``` **Authentication for Uploads:** For uploading results, you need a HuggingFace token with write permissions to the target dataset. You can provide the token in several ways (in order of precedence): 1. Command line: `--token hf_your_token_here` 3. Environment variable: `HF_TOKEN` ### Running Specific Benchmarks ```bash # Include only specific benchmarks python run_benchmarks.py --include llama # Exclude specific benchmarks python run_benchmarks.py --exclude old_benchmark ## Output Format Results are saved as JSON files with the following structure: ```json { "model_name": "llama_2_7b", "benchmark_scenarios": [ { "scenario_name": "eager_variant", "metadata": { "timestamp": "2025-01-XX...", "commit_id": "abc123...", "hardware_info": { "gpu_name": "NVIDIA A100", "gpu_memory_total": 40960, "cpu_count": 64 }, "config": { "variant": "eager", "warmup_iterations": 3, "measurement_iterations": 5 } }, "measurements": { "latency": { "mean": 2.45, "median": 2.43, "std": 0.12, "min": 2.31, "max": 2.67, "p95": 2.61, "p99": 2.65 }, "time_to_first_token": { "mean": 0.15, "std": 0.02 }, "tokens_per_second": { "mean": 87.3, "unit": "tokens/sec" } }, "gpu_metrics": { "gpu_utilization_mean": 85.2, "gpu_memory_used_mean": 12450 } } ] } ``` ### Debug Mode ```bash python run_benchmarks.py --log-level DEBUG ``` ## Contributing To add new benchmarks: 1. Create a new file in `benches/` 2. Implement the `ModelBenchmark` interface 3. Add a runner function (`run_` or `run_benchmark`) 4. run_benchmarks.py --- ### Overview (`docker/README.md`) # Dockers for `transformers` In this folder you will find various docker files, and some subfolders. - dockerfiles (ex: `consistency.dockerfile`) present under `~/docker` are used for our "fast" CIs. You should be able to use them for tasks that only need CPU. For example `torch-light` is a very light weights container (703MiB). They all build on top of `ci-base.dockerfile`, using the Python version from the repo's `.python-version`. Build them with `docker buildx bake` from the repository root, e.g. `PYTHON_VERSION=$(cat .python-version) docker buildx bake -f docker/docker-bake.hcl torch-light` (a plain `docker build` won't resolve the `ci-base` stage). - subfolders contain dockerfiles used for our `slow` CIs, which *can* be used for GPU tasks, but they are **BIG** as they were not specifically designed for a single model / single task. Thus the `~/docker/transformers-pytorch-gpu` includes additional dependencies to allow us to run ALL model tests (say `librosa` or `tesseract`, which you do not need to run LLMs) The GPU ones share the steps in `gpu-common/` (Python from the repo's `.python-version`, finishing the `transformers` install, checking the pinned torch) through a named build context, e.g. `docker build --build-context gpu-common=docker/gpu-common --build-arg PYTHON_VERSION=$(cat .python-version) docker/transformers-all-latest-gpu`. Note that in both case, you need to run `uv pip install -e .`, which should take around 5 seconds. We do it outside the dockerfile for the need of our CI: we checkout a new branch each time, and the `transformers` code is thus updated. We are open to contribution, and invite the community to create dockerfiles with potential arguments that properly choose extras depending on the model's dependencies! :hugs: --- ### Overview (`docs/README.md`) # Writing docs The Transformers docs live under `docs/source//` and are built with [doc-builder](https://github.com/huggingface/doc-builder). > [!TIP] > You usually don't need to build the docs locally. When you open a PR that touches files under `docs/`, a bot builds a preview and posts the link as a comment on the PR. Build locally for a faster iteration loop, or while drafting changes before you open a PR. ## Building and previewing locally Install the quality dependencies and [doc-builder](https://github.com/huggingface/doc-builder) from the root of the repository. The `[quality]` extra is enough for docs-only work. Use `[dev]` if your change also needs the full development dependency set. ```bash pip install -e ".[quality]" pip install git+https://github.com/huggingface/doc-builder ``` To build the Markdown files into a temporary folder you can inspect in any Markdown editor: ```bash doc-builder build transformers docs/source/en/ --build_dir ~/tmp/test-build ``` To live-preview the docs in a browser at [http://localhost:5173](http://localhost:5173), install [watchdog](https://github.com/gorakhargosh/watchdog) and run `preview`: ```bash pip install watchdog doc-builder preview transformers docs/source/en/ ``` > [!WARNING] > `preview` only picks up files that exist when it starts. After you add a brand-new page, update `_toctree.yml` and restart `preview`. Don't commit the built output. Only changes under `docs/source/` are reviewed. ## Adding a new file to the docs Pages live in `docs/source//` as Markdown (`.md`) files. The sidebar navigation lives in [_toctree.yml](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml), and a new page only shows up in the sidebar after you add it there. 1. Create the Markdown file under `docs/source/en/` (or the appropriate language directory). Match the file naming and license header of an existing page. Copying a similar page is the fastest way to start. 2. Add an entry to `docs/source/en/_toctree.yml` that points to the filename without the `.md` extension. Each entry has two fields: - `local`: the file path relative to `docs/source//`, without the extension. - `title`: the human-readable label that appears in the sidebar. For a page nested inside a subsection, add it to the inner `sections` list: ```yaml - isExpanded: false sections: - local: contributing title: Contribute to Transformers - local: my_new_contributor_guide title: My new contributor guide title: Contribute ``` ## Keeping code snippets device agnostic Readers run our snippets on NVIDIA GPUs, AMD ROCm, Intel XPU, Apple MPS, Ascend NPU and CPU, so avoid hard-coding `"cuda"`. - Use `device_map="auto"` only when automatic dispatch is intended. Keep explicit `cuda` in CUDA-specific examples. - Move inputs with `inputs.to(model.device)` instead of `.to("cuda")` or `.cuda()`. - Use `torch.accelerator.synchronize()` for synchronization and and `torch.Event(enable_timing=True)` for timing. Memory helpers live under `torch.accelerator.memory`, for example `torch.accelerator.memory.empty_cache()` and `torch.accelerator.memory.max_memory_allocated()`. - If you really need a device string, use `device = torch.accelerator.current_accelerator().type if torch.accelerator.is_available() else "cpu"`. - Keep `cuda` only where it is genuinely CUDA-specific: toolchain instructions (`nvcc`, `nvidia-smi`), NVIDIA-only backends, PyTorch API names such as `use_cuda_graph`, and sample outputs copied from a real run. ## doc-builder syntax doc-builder accepts standard Markdown plus a few extensions you'll see across our docs. For the full set of supported syntax (inference snippets, file-include blocks, redirects, multilingual builds, etc.), see the [doc-builder README](https://github.com/huggingface/doc-builder#writing-documentation-for-hugging-face-libraries). ### Tips and warnings Use GitHub-style blockquote callouts for notes, tips, and warnings: ```md > [!TIP] > Use `device_map="auto"` to let Transformers place model shards across available devices. > [!WARNING] > `from_pretrained` downloads the full checkpoint on first use. Set `cache_dir` to control where it lands. ``` Older pages may use the legacy `` component. Prefer blockquotes for new content. ### Internal links to classes and functions Wrap a class, function, or method name in square brackets and backticks to link to its doc page. doc-builder resolves the link automatically: ```md Use [`AutoModel`] to load a model from a checkpoint, then call [`~PreTrainedModel.from_pretrained`]. ``` A few variations: - Prefix the name with `~` to render only the last component (`from_pretrained`) instead of the full path (`PreTrainedModel.from_pretrained`). - For objects nested in a submodule, include the path inside the backticks, like `utils.ModelOutput`. - The same syntax links to objects in other Hugging Face libraries, e.g. `accelerate.Accelerator`. ### Tabbed options To show alternative snippets (CLI vs. Python, different backends, etc.) as tabs, use ``: ````md ```bash pip install transformers ``` ```bash uv pip install transformers ``` ```` ### Auto-generated API reference > [!IMPORTANT] > Always leave a blank line after `[[autodoc]]` so the CI checks pass. Use `[[autodoc]]` to render the docstring of a class or function. The marker pulls the description, arguments, and (for classes) every public method: ```md ## AutoModel [[autodoc]] AutoModel ``` To restrict the output to specific methods, list them as a bulleted sub-list: ```md [[autodoc]] BertTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask ``` To pull in a method that isn't documented by default (for example, `__call__`), start the list with `all` and then add the extras: ```md [[autodoc]] BertTokenizer - all - __call__ ``` ### Testable code blocks Tag Python fences with `runnable` (optionally followed by `: