### README # Generating the documentation To generate the documentation, you have to build it. Several packages are necessary to build the doc. First, you need to install the project itself by running the following command at the root of the code repository: ```bash pip install -e . ``` You also need to install 2 extra packages: ```bash # `hf-doc-builder` to build the docs pip install git+https://github.com/huggingface/doc-builder@main # `watchdog` for live reloads pip install watchdog ``` --- **NOTE** You only need to generate the documentation to inspect it locally (if you're planning changes and want to check how they look before committing for instance). You don't have to commit the built documentation. --- ## Building the documentation Once you have setup the `doc-builder` and additional packages with the pip install command above, you can generate the documentation by typing the following command: ```bash doc-builder build autotrain docs/source/ --build_dir ~/tmp/test-build ``` You can adapt the `--build_dir` to set any temporary folder that you prefer. This command will create it and generate the MDX files that will be rendered as the documentation on the main website. You can inspect them in your favorite Markdown editor. ## Previewing the documentation To preview the docs, run the following command: ```bash doc-builder preview autotrain docs/source/ ``` The docs will be viewable at [http://localhost:5173](http://localhost:5173). You can also preview the docs once you have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives. --- **NOTE** The `preview` command only works with existing doc files. When you add a completely new file, you need to update `_toctree.yml` & restart `preview` command (`ctrl-c` to stop it & call `doc-builder preview ...` again). --- --- ### Source/Tasks/Extractive Qa # Extractive Question Answering with AutoTrain Extractive Question Answering (QA) enables AI models to find and extract precise answers from text passages. This guide shows you how to train custom QA models using AutoTrain, supporting popular architectures like BERT, RoBERTa, and DeBERTa. ## What is Extractive Question Answering? Extractive QA models learn to: - Locate exact answer spans within longer text passages - Understand questions and match them to relevant context - Extract precise answers rather than generating them - Handle both simple and complex queries about the text ## Preparing your Data Your dataset needs these essential columns: - `text`: The passage containing potential answers (also called context) - `question`: The query you want to answer - `answer`: Answer span information including text and position Here is an example of how your dataset should look: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Note: the preferred format for question answering is JSONL, if you want to use CSV, the `answer` column should be stringified JSON with the keys `text` and `answer_start`. Example dataset from Hugging Face Hub: [lhoestq/squad](https://huggingface.co/datasets/lhoestq/squad) P.S. You can use both squad and squad v2 data format with correct column mappings. ## Training Options ### Local Training Train models on your own hardware with full control over the process. To train an Extractive QA model locally, you need a config file: ```yaml task: extractive-qa base_model: google-bert/bert-base-uncased project_name: autotrain-bert-ex-qa1 log: tensorboard backend: local data: path: lhoestq/squad train_split: train valid_split: validation column_mapping: text_column: context question_column: question answer_column: answers params: max_seq_length: 512 max_doc_stride: 128 epochs: 3 batch_size: 4 lr: 2e-5 optimizer: adamw_torch scheduler: linear gradient_accumulation: 1 mixed_precision: fp16 hub: username: ${HF_USERNAME} token: ${HF_TOKEN} push_to_hub: true ``` To train the model, run the following command: ```bash $ autotrain --config config.yaml ``` Here, we are training a BERT model on the SQuAD dataset using the Extractive QA task. The model is trained for 3 epochs with a batch size of 4 and a learning rate of 2e-5. The training process is logged using TensorBoard. The model is trained locally and pushed to the Hugging Face Hub after training. ### Cloud Training on Hugging Face Train models using Hugging Face's cloud infrastructure for better scalability. As always, pay special attention to column mapping. ## Parameter Reference [[autodoc]] trainers.extractive_question_answering.params.ExtractiveQuestionAnsweringParams --- ### Source/Tasks/Image Classification Regression # Image Classification & Regression Image classification is a form of supervised learning where a model is trained to identify and categorize objects within images. AutoTrain simplifies the process, enabling you to train a state-of-the-art image classification model by simply uploading labeled example images. Image regression/scoring is a form of supervised learning where a model is trained to predict a score or value for an image. AutoTrain simplifies the process, enabling you to train a state-of-the-art image scoring model by simply uploading labeled example images. ## Preparing your data To ensure your image classification model trains effectively, follow these guidelines for preparing your data: ### Organizing Images For Image Classification Prepare a zip file containing your categorized images. Each category should have its own subfolder named after the class it represents. For example, to differentiate between 'cats' and 'dogs', your zip file structure should resemble the following: ``` cats_and_dogs.zip ├── cats │ ├── cat.1.jpg │ ├── cat.2.jpg │ ├── cat.3.jpg │ └── ... └── dogs ├── dog.1.jpg ├── dog.2.jpg ├── dog.3.jpg └── ... ``` You can also use a dataset from the Hugging Face Hub. Example dataset from Hugging Face Hub: [truepositive/hotdog_nothotdog](https://huggingface.co/datasets/truepositive/hotdog_nothotdog). ### Organizing Images for Image Regression/Scoring Prepare a zip file containing your images and metadata.jsonl. ``` Archive.zip ├── 0001.png ├── 0002.png ├── 0003.png ├── . ├── . ├── . └── metadata.jsonl ``` Example for `metadata.jsonl`: ``` {"file_name": "0001.png", "target": 0.5} {"file_name": "0002.png", "target": 0.7} {"file_name": "0003.png", "target": 0.3} ``` Please note that metadata.jsonl should contain the `file_name` and the `target` value for each image. You can also use a dataset from the Hugging Face Hub. Example dataset from Hugging Face Hub: [abhishek/img-quality-full](https://huggingface.co/datasets/abhishek/img-quality-full). ### Image Requirements - Format: Ensure all images are in JPEG, JPG, or PNG format. - Quantity: Include at least 5 images per class to provide the model with sufficient examples for learning. - Exclusivity: The zip file should exclusively contain folders named after the classes, and these folders should only contain relevant images. No additional files or nested folders should be included. ** Additional Tips** - Uniformity: While not required, having images of similar sizes and resolutions can help improve model performance. - Variability: Include a variety of images for each class to encompass the range of appearances and contexts the model might encounter in real-world scenarios. Some points to keep in mind: - The zip file should contain multiple folders (the classes), each folder should contain images of a single class. - The name of the folder should be the name of the class. - The images must be jpeg, jpg or png. - There should be at least 5 images per class. - There must not be any other files in the zip file. - There must not be any other folders inside the zip folder. When train.zip is decompressed, it creates two folders: cats and dogs. these are the two categories for classification. The images for both categories are in their respective folders. You can have as many categories as you want. ## Column Mapping For image classification, if you are using a `zip` dataset format, the column mapping should be default and should not be changed. ```yaml data: . . . column_mapping: image_column: image target_column: label ``` For image regression, the column mapping must be as follows: ```yaml data: . . . column_mapping: image_column: image target_column: target ``` For image regression, `metadata.jsonl` should contain the `file_name` and the `target` value for each image. If you are using a dataset from the Hugging Face Hub, you should set appropriate column mappings based on the dataset. ## Training ### Local Training To train the model locally, create a configuration file (config.yaml) with the following content: ```yaml task: image_classification base_model: google/vit-base-patch16-224 project_name: autotrain-cats-vs-dogs-finetuned log: tensorboard backend: local data: path: cats_vs_dogs train_split: train valid_split: null column_mapping: image_column: image target_column: label params: epochs: 2 batch_size: 4 lr: 2e-5 optimizer: adamw_torch scheduler: linear gradient_accumulation: 1 mixed_precision: fp16 hub: username: ${HF_USERNAME} token: ${HF_TOKEN} push_to_hub: true ``` Here, we are using `cats_and_dogs` dataset from Hugging Face Hub. The model is trained for 2 epochs with a batch size of 4 and a learning rate of `2e-5`. We are using the `adamw_torch` optimizer and the `linear` scheduler. We are also using mixed precision training with a gradient accumulation of 1. In order to use a local dataset, you can change the `data` section to: ```yaml data: path: data/ train_split: train # this folder inside data/ will be used for training, it contains the images in subfolders. valid_split: valid # this folder inside data/ will be used for validation, it contains the images in subfolders. can also be null. column_mapping: image_column: image target_column: label ``` Similarly, for image regression, you can use the following configuration file: ```yaml task: image_regression base_model: microsoft/resnet-50 project_name: autotrain-img-quality-resnet50 log: tensorboard backend: local data: path: abhishek/img-quality-full train_split: train valid_split: null column_mapping: image_column: image target_column: target params: epochs: 10 batch_size: 8 lr: 2e-3 optimizer: adamw_torch scheduler: cosine gradient_accumulation: 1 mixed_precision: fp16 hub: username: ${HF_USERNAME} token: ${HF_TOKEN} push_to_hub: true ``` To train the model, run the following command: ```bash $ autotrain --config config.yaml ``` This will start the training process and save the model to the Hugging Face Hub after training is complete. In case you dont want to save the model to the hub, you can set `push_to_hub` to `false` in the configuration file. ### Training on Hugging Face Spaces To train the model on Hugging Face Spaces, create a training space as described in `Quickstart` section. An example UI for training an image scoring model on Hugging Face Spaces is shown below: In this example, we are training an image scoring model using the `microsoft/resnet-50` model on the `abhishek/img-quality-full` dataset. We are training the model for 3 epochs with a batch size of 8 and a learning rate of `5e-5`. We are using the `adamw_torch` optimizer and the `linear` scheduler. We are also using mixed precision training with a gradient accumulation of 1. Note how the column mapping has now been changed and `target` points to `quality_mos` column in the dataset. To train the model, click on the `Start Training` button. This will start the training process and save the model to the Hugging Face Hub after training is complete. ## Parameters ### Image Classification Parameters [[autodoc]] trainers.image_classification.params.ImageClassificationParams ### Image Regression Parameters [[autodoc]] trainers.image_regression.params.ImageRegressionParams --- ### Source/Tasks/Llm Finetuning # LLM Finetuning with AutoTrain Advanced AutoTrain Advanced makes it easy to fine-tune large language models (LLMs) for your specific use cases. This guide covers everything you need to know about LLM fine-tuning. ## Key Features - Simple data preparation with CSV and JSONL formats - Support for multiple training approaches (SFT, DPO, ORPO) - Built-in chat templates - Local and cloud training options - Optimized training parameters ## Supported Training Methods AutoTrain supports multiple specialized trainers: - `llm`: Generic LLM trainer - `llm-sft`: Supervised Fine-Tuning trainer - `llm-reward`: Reward modeling trainer - `llm-dpo`: Direct Preference Optimization trainer - `llm-orpo`: ORPO (Optimal Reward Policy Optimization) trainer ## Data Preparation LLM finetuning accepts data in CSV and JSONL formats. JSONL is the preferred format. How data is formatted depends on the task you are training the LLM for. ### Classic Text Generation For text generation, the data should be in the following format: | text | |---------------------------------------------------------------| | wikipedia is a free online encyclopedia | | it is a collaborative project | | that anyone can edit | | wikipedia is the largest and most popular general reference work on the internet | An example dataset for this format can be found here: [stas/openwebtext-10k](https://huggingface.co/datasets/stas/openwebtext-10k) Example tasks: - Text generation - Code completion Compatible trainers: - SFT Trainer - Generic Trainer ### Chatbot / question-answering / code generation / function calling For this task, you can use CSV or JSONL data. If you are formatting the data yourself (adding start, end tokens, etc.), you can use CSV or JSONL format. If you do not want to format the data yourself and want `--chat-template` parameter to format the data for you, you must use JSONL format. In both cases, CSV and JSONL can be used interchangeably but JSONL is the most preferred format. To train a chatbot, your data will have `content` and `role`. Some models support `system` role as well. Here is an example of a chatbot dataset (single sample): ``` [{'content': 'Help write a letter of 100 -200 words to my future self for ' 'Kyra, reflecting on her goals and aspirations.', 'role': 'user'}, {'content': 'Dear Future Self,\n' '\n' "I hope you're happy and proud of what you've achieved. As I " "write this, I'm excited to think about our goals and how far " "you've come. One goal was to be a machine learning engineer. I " "hope you've worked hard and become skilled in this field. Keep " 'learning and innovating. Traveling was important to us. I hope ' "you've seen different places and enjoyed the beauty of our " 'world. Remember the memories and lessons. Starting a family ' 'mattered to us. If you have kids, treasure every moment. Be ' 'patient, loving, and grateful for your family.\n' '\n' 'Take care of yourself. Rest, reflect, and cherish the time you ' 'spend with loved ones. Remember your dreams and celebrate what ' "you've achieved. Your determination brought you here. I'm " "excited to see the person you've become, the impact you've made, " 'and the love and joy in your life. Embrace opportunities and ' 'keep dreaming big.\n' '\n' 'With love,\n' 'Kyra', 'role': 'assistant'}] ``` As you can see, the data has `content` and `role` columns. The `role` column can be `user` or `assistant` or `system`. This data is, however, not formatted for training. You can use the `--chat-template` parameter to format the data during training. `--chat-template` supports the following kinds of templates: - `none` (default) - `zephyr` - `chatml` - `tokenizer`: use chat template mentioned in tokenizer config A multi-line sample is also shown below: ```json [{"content": "hello", "role": "user"}, {"content": "hi nice to meet you", "role": "assistant"}] [{"content": "how are you", "role": "user"}, {"content": "I am fine", "role": "assistant"}] [{"content": "What is your name?", "role": "user"}, {"content": "My name is Mary", "role": "assistant"}] [{"content": "Which is the best programming language?", "role": "user"}, {"content": "Python", "role": "assistant"}] . . . ``` An example dataset for this format can be found here: [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) If you dont want to format the data using `--chat-template`, you can format the data yourself and use the following format: ``` <|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHelp write a letter of 100 -200 words to my future self for Kyra, reflecting on her goals and aspirations.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nDear Future Self,\n\nI hope you're happy and proud of what you've achieved. As I write this, I'm excited to think about our goals and how far you've come. One goal was to be a machine learning engineer. I hope you've worked hard and become skilled in this field. Keep learning and innovating. Traveling was important to us. I hope you've seen different places and enjoyed the beauty of our world. Remember the memories and lessons. Starting a family mattered to us. If you have kids, treasure every moment. Be patient, loving, and grateful for your family.\n\nTake care of yourself. Rest, reflect, and cherish the time you spend with loved ones. Remember your dreams and celebrate what you've achieved. Your determination brought you here. I'm excited to see the person you've become, the impact you've made, and the love and joy in your life. Embrace opportunities and keep dreaming big.\n\nWith love,\nKyra<|eot_id|> ``` A sample multi-line dataset is shown below: ```json [{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nhello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nhi nice to meet you<|eot_id|>"}] [{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nhow are you<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nI am fine<|eot_id|>"}] [{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is your name?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nMy name is Mary<|eot_id|>"}] [{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhich is the best programming language?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nPython<|eot_id|>"}] . . . ``` An example dataset for this format can be found here: [timdettmers/openassistant-guanaco](https://huggingface.co/datasets/timdettmers/openassistant-guanaco) In the examples above, we have seen only two turns: one from the user and one from the assistant. However, you can have multiple turns from the user and assistant in a single sample. Chat models can be trained using the following trainers: - SFT Trainer: - requires only `text` column - example dataset: [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) - Generic Trainer: - requires only `text` column - example dataset: [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) - Reward Trainer: - requires `text` and `rejected_text` columns - example dataset: [trl-lib/ultrafeedback_binarized](https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized) - DPO Trainer: - requires `prompt`, `text`, and `rejected_text` columns - example dataset: [trl-lib/ultrafeedback_binarized](https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized) - ORPO Trainer: - requires `prompt`, `text`, and `rejected_text` columns - example dataset: [trl-lib/ultrafeedback_binarized](https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized) The only difference between the data format for reward trainer and DPO/ORPO trainer is that the reward trainer requires only `text` and `rejected_text` columns, while the DPO/ORPO trainer requires an additional `prompt` column. ## Best Practices for LLM Fine-tuning ### Memory Optimization - Use appropriate `block_size` and `model_max_length` for your hardware - Enable mixed precision training when possible - Utilize PEFT techniques for large models ### Data Quality - Clean and validate your training data - Ensure balanced conversation samples - Use appropriate chat templates ### Training Tips - Start with small learning rates - Monitor training metrics using tensorboard - Validate model outputs during training ### Related Resources - [AutoTrain Documentation](https://huggingface.co/docs/autotrain) - [Example Fine-tuned Models](https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads) - [Training Datasets](https://huggingface.co/datasets?task_categories=task_categories:text-generation) ## Training ### Local Training Locally the training can be performed by using `autotrain --config config.yaml` command. The `config.yaml` file should contain the following parameters: ```yaml task: llm-orpo base_model: meta-llama/Meta-Llama-3-8B-Instruct project_name: autotrain-llama3-8b-orpo log: tensorboard backend: local data: path: argilla/distilabel-capybara-dpo-7k-binarized train_split: train valid_split: null chat_template: chatml column_mapping: text_column: chosen rejected_text_column: rejected prompt_text_column: prompt params: block_size: 1024 model_max_length: 8192 max_prompt_length: 512 epochs: 3 batch_size: 2 lr: 3e-5 peft: true quantization: int4 target_modules: all-linear padding: right optimizer: adamw_torch scheduler: linear gradient_accumulation: 4 mixed_precision: fp16 hub: username: ${HF_USERNAME} token: ${HF_TOKEN} push_to_hub: true ``` In the above config file, we are training a model using the ORPO trainer. The model is trained on the `meta-llama/Meta-Llama-3-8B-Instruct` model. The data is `argilla/distilabel-capybara-dpo-7k-binarized` dataset. The `chat_template` parameter is set to `chatml`. The `column_mapping` parameter is used to map the columns in the dataset to the required columns for the ORPO trainer. The `params` section contains the training parameters such as `block_size`, `model_max_length`, `epochs`, `batch_size`, `lr`, `peft`, `quantization`, `target_modules`, `padding`, `optimizer`, `scheduler`, `gradient_accumulation`, and `mixed_precision`. The `hub` section contains the username and token for the Hugging Face account and the `push_to_hub` parameter is set to `true` to push the trained model to the Hugging Face Hub. If you have training file locally, you can change data part to: ```yaml data: path: path/to/training/file train_split: train # name of the training file valid_split: null chat_template: chatml column_mapping: text_column: chosen rejected_text_column: rejected prompt_text_column: prompt ``` The above assumes you have `train.csv` or `train.jsonl` in the `path/to/training/file` directory and you will be applying `chatml` template to the data. You can run the training using the following command: ```bash $ autotrain --config config.yaml ``` More example config files for finetuning different types of lllm and different tasks can be found in the [here](https://github.com/huggingface/autotrain-advanced/tree/main/configs/llm_finetuning). ### Training in Hugging Face Spaces If you are training in Hugging Face Spaces, everything is the same as local training: In the UI, you need to make sure you select the right model, the dataset and the splits. Special care should be taken for `column_mapping`. Once you are happy with the parameters, you can click on the `Start Training` button to start the training process. ## Parameters ### LLM Fine Tuning Parameters [[autodoc]] trainers.clm.params.LLMTrainingParams ### Task specific parameters The length parameters used for different trainers can be different. Some require more context than others. - block_size: This is the maximum sequence length or length of one block of text. Setting to -1 determines block size automatically. Default is -1. - model_max_length: Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage. Default is 1024 - max_prompt_length: Specify the maximum length for prompts used in training, particularly relevant for tasks requiring initial contextual input. Used only for `orpo` and `dpo` trainer. - max_completion_length: Completion length to use, for orpo: encoder-decoder models only. For dpo, it is the length of the completion text. **NOTE**: - block size cannot be greater than model_max_length! - max_prompt_length cannot be greater than model_max_length! - max_prompt_length cannot be greater than block_size! - max_completion_length cannot be greater than model_max_length! - max_completion_length cannot be greater than block_size! **NOTE**: Not following these constraints will result in an error / nan losses. #### Generic Trainer ``` --add_eos_token, --add-eos-token Toggle whether to automatically add an End Of Sentence (EOS) token at the end of texts, which can be critical for certain types of models like language models. Only used for `default` trainer --block_size BLOCK_SIZE, --block-size BLOCK_SIZE Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to -1 determines block size automatically. Default is -1. --model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage. Default is 1024 ``` #### SFT Trainer ``` --block_size BLOCK_SIZE, --block-size BLOCK_SIZE Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to -1 determines block size automatically. Default is -1. --model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage. Default is 1024 ``` #### Reward Trainer ``` --block_size BLOCK_SIZE, --block-size BLOCK_SIZE Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to -1 determines block size automatically. Default is -1. --model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage. Default is 1024 ``` #### DPO Trainer ``` --dpo-beta DPO_BETA, --dpo-beta DPO_BETA Beta for DPO trainer --model-ref MODEL_REF Reference model to use for DPO when not using PEFT --block_size BLOCK_SIZE, --block-size BLOCK_SIZE Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to -1 determines block size automatically. Default is -1. --model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage. Default is 1024 --max_prompt_length MAX_PROMPT_LENGTH, --max-prompt-length MAX_PROMPT_LENGTH Specify the maximum length for prompts used in training, particularly relevant for tasks requiring initial contextual input. Used only for `orpo` trainer. --max_completion_length MAX_COMPLETION_LENGTH, --max-completion-length MAX_COMPLETION_LENGTH Completion length to use, for orpo: encoder-decoder models only ``` #### ORPO Trainer ``` --block_size BLOCK_SIZE, --block-size BLOCK_SIZE Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to -1 determines block size automatically. Default is -1. --model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage. Default is 1024 --max_prompt_length MAX_PROMPT_LENGTH, --max-prompt-length MAX_PROMPT_LENGTH Specify the maximum length for prompts used in training, particularly relevant for tasks requiring initial contextual input. Used only for `orpo` trainer. --max_completion_length MAX_COMPLETION_LENGTH, --max-completion-length MAX_COMPLETION_LENGTH Completion length to use, for orpo: encoder-decoder models only ``` --- ### Source/Tasks/Object Detection # Object Detection Object detection is a form of supervised learning where a model is trained to identify and categorize objects within images. AutoTrain simplifies the process, enabling you to train a state-of-the-art object detection model by simply uploading labeled example images. ## Preparing your data To ensure your object detection model trains effectively, follow these guidelines for preparing your data: ### Organizing Images Prepare a zip file containing your images and metadata.jsonl. ``` Archive.zip ├── 0001.png ├── 0002.png ├── 0003.png ├── . ├── . ├── . └── metadata.jsonl ``` Example for `metadata.jsonl`: ``` {"file_name": "0001.png", "objects": {"bbox": [[302.0, 109.0, 73.0, 52.0]], "category": [0]}} {"file_name": "0002.png", "objects": {"bbox": [[810.0, 100.0, 57.0, 28.0]], "category": [1]}} {"file_name": "0003.png", "objects": {"bbox": [[160.0, 31.0, 248.0, 616.0], [741.0, 68.0, 202.0, 401.0]], "category": [2, 2]}} ``` Please note that bboxes need to be in COCO format `[x, y, width, height]`. ### Image Requirements - Format: Ensure all images are in JPEG, JPG, or PNG format. - Quantity: Include at least 5 images to provide the model with sufficient examples for learning. - Exclusivity: The zip file should exclusively contain images and metadata.jsonl. No additional files or nested folders should be included. Some points to keep in mind: - The images must be jpeg, jpg or png. - There should be at least 5 images per split. - There must not be any other files in the zip file. - There must not be any other folders inside the zip folder. When train.zip is decompressed, it creates no folders: only images and metadata.jsonl. ## Parameters [[autodoc]] trainers.object_detection.params.ObjectDetectionParams --- ### Source/Tasks/Sentence Transformer # Sentence Transformers This task lets you easily train or fine-tune a Sentence Transformer model on your own dataset. AutoTrain supports the following types of sentence transformer finetuning: - `pair`: dataset with two sentences: anchor and positive - `pair_class`: dataset with two sentences: premise and hypothesis and a target label - `pair_score`: dataset with two sentences: sentence1 and sentence2 and a target score - `triplet`: dataset with three sentences: anchor, positive and negative - `qa`: dataset with two sentences: query and answer ## Data Format Sentence Transformers finetuning accepts data in CSV/JSONL format. You can also use a dataset from Hugging Face Hub. ### `pair` For `pair` training, the data should be in the following format: | anchor | positive | |--------|----------| | hello | hi | | how are you | I am fine | | What is your name? | My name is Abhishek | | Which is the best programming language? | Python | ### `pair_class` For `pair_class` training, the data should be in the following format: | premise | hypothesis | label | |---------|------------|-------| | hello | hi | 1 | | how are you | I am fine | 0 | | What is your name? | My name is Abhishek | 1 | | Which is the best programming language? | Python | 1 | ### `pair_score` For `pair_score` training, the data should be in the following format: | sentence1 | sentence2 | score | |-----------|-----------|-------| | hello | hi | 0.8 | | how are you | I am fine | 0.2 | | What is your name? | My name is Abhishek | 0.9 | | Which is the best programming language? | Python | 0.7 | ### `triplet` For `triplet` training, the data should be in the following format: | anchor | positive | negative | |--------|----------|----------| | hello | hi | bye | | how are you | I am fine | I am not fine | | What is your name? | My name is Abhishek | Whats it to you? | | Which is the best programming language? | Python | Javascript | ### `qa` For `qa` training, the data should be in the following format: | query | answer | |-------|--------| | hello | hi | | how are you | I am fine | | What is your name? | My name is Abhishek | | Which is the best programming language? | Python | ## Parameters [[autodoc]] trainers.sent_transformers.params.SentenceTransformersParams --- ### Source/Tasks/Seq2seq # Seq2Seq Seq2Seq is a task that involves converting a sequence of words into another sequence of words. It is used in machine translation, text summarization, and question answering. ## Data Format You can have the dataset as a CSV file: ```csv text,target "this movie is great","dieser Film ist großartig" "this movie is bad","dieser Film ist schlecht" . . . ``` Or as a JSONL file: ```json {"text": "this movie is great", "target": "dieser Film ist großartig"} {"text": "this movie is bad", "target": "dieser Film ist schlecht"} . . . ``` ## Columns Your CSV/JSONL dataset must have two columns: `text` and `target`. ## Parameters [[autodoc]] trainers.seq2seq.params.Seq2SeqParams --- ### Source/Tasks/Tabular # Tabular Classification / Regression Using AutoTrain, you can train a model to classify or regress tabular data easily. All you need to do is select from a list of models and upload your dataset. Parameter tuning is done automatically. ## Models The following models are available for tabular classification / regression. - xgboost - random_forest - ridge - logistic_regression - svm - extra_trees - gradient_boosting - adaboost - decision_tree - knn ## Data Format ```csv id,category1,category2,feature1,target 1,A,X,0.3373961604172684,1 2,B,Z,0.6481718720511972,0 3,A,Y,0.36824153984054797,1 4,B,Z,0.9571551589530464,1 5,B,Z,0.14035078041264515,1 6,C,X,0.8700872583584364,1 7,A,Y,0.4736080452737105,0 8,C,Y,0.8009107519796442,1 9,A,Y,0.5204774795512048,0 10,A,Y,0.6788795301189603,0 . . . ``` ## Columns Your CSV dataset must have two columns: `id` and `target`. ## Parameters [[autodoc]] trainers.tabular.params.TabularParams --- ### Source/Tasks/Text Classification Regression # Text Classification & Regression Training a text classification/regression model with AutoTrain is super-easy! Get your data ready in proper format and then with just a few clicks, your state-of-the-art model will be ready to be used in production. Config file task names: - `text_classification` - `text-classification` - `text_regression` - `text-regression` ## Data Format Text classification/regression supports datasets in both CSV and JSONL formats. ### CSV Format Let's train a model for classifying the sentiment of a movie review. The data should be in the following CSV format: ```csv text,target "this movie is great",positive "this movie is bad",negative . . . ``` As you can see, we have two columns in the CSV file. One column is the text and the other is the label. The label can be any string. In this example, we have two labels: `positive` and `negative`. You can have as many labels as you want. And if you would like to train a model for scoring a movie review on a scale of 1-5. The data can be as follows: ```csv text,target "this movie is great",4.9 "this movie is bad",1.5 . . . ``` ### JSONL Format Instead of CSV you can also use JSONL format. The JSONL format should be as follows: ```json {"text": "this movie is great", "target": "positive"} {"text": "this movie is bad", "target": "negative"} . . . ``` and for regression: ```json {"text": "this movie is great", "target": 4.9} {"text": "this movie is bad", "target": 1.5} . . ``` ### Column Mapping / Names Your CSV dataset must have two columns: `text` and `target`. If your column names are different than `text` and `target`, you can map the dataset column to AutoTrain column names. ## Training ### Local Training To train a text classification/regression model locally, you can use the `autotrain --config config.yaml` command. Here is an example of a `config.yaml` file for training a text classification model: ```yaml task: text_classification # or text_regression base_model: google-bert/bert-base-uncased project_name: autotrain-bert-imdb-finetuned log: tensorboard backend: local data: path: stanfordnlp/imdb train_split: train valid_split: test column_mapping: text_column: text target_column: label params: max_seq_length: 512 epochs: 3 batch_size: 4 lr: 2e-5 optimizer: adamw_torch scheduler: linear gradient_accumulation: 1 mixed_precision: fp16 hub: username: ${HF_USERNAME} token: ${HF_TOKEN} push_to_hub: true ``` In this example, we are training a text classification model using the `google-bert/bert-base-uncased` model on the IMDB dataset. We are using the `stanfordnlp/imdb` dataset, which is already available on Hugging Face Hub. We are training the model for 3 epochs with a batch size of 4 and a learning rate of `2e-5`. We are using the `adamw_torch` optimizer and the `linear` scheduler. We are also using mixed precision training with a gradient accumulation of 1. If you want to use a local CSV/JSONL dataset, you can change the `data` section to: ```yaml data: path: data/ # this must be the path to the directory containing the train and valid files train_split: train # this must be either train.csv or train.json valid_split: valid # this must be either valid.csv or valid.json column_mapping: text_column: text # this must be the name of the column containing the text target_column: label # this must be the name of the column containing the target ``` To train the model, run the following command: ```bash $ autotrain --config config.yaml ``` You can find example config files for text classification and regression in the [here](https://github.com/huggingface/autotrain-advanced/tree/main/configs/text_classification) and [here](https://github.com/huggingface/autotrain-advanced/tree/main/configs/text_regression) respectively. ### Training on Hugging Face Spaces The parameters for training on Hugging Face Spaces are the same as for local training. If you are using your own dataset, select "Local" as dataset source and upload your dataset. In the following screenshot, we are training a text classification model using the `google-bert/bert-base-uncased` model on the IMDB dataset. For text regression, all you need to do is select "Text Regression" as the task and everything else remains the same (except the data, of course). ## Training Parameters Training parameters for text classification and regression are the same. [[autodoc]] trainers.text_classification.params.TextClassificationParams --- ### Source/Tasks/Token Classification # Token Classification Token classification is the task of classifying each token in a sequence. This can be used for Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and more. Get your data ready in proper format and then with just a few clicks, your state-of-the-art model will be ready to be used in production. ## Data Format The data should be in the following CSV format: ```csv tokens,tags "['I', 'love', 'Paris']","['O', 'O', 'B-LOC']" "['I', 'live', 'in', 'New', 'York']","['O', 'O', 'O', 'B-LOC', 'I-LOC']" . . . ``` or you can also use JSONL format: ```json {"tokens": ["I", "love", "Paris"],"tags": ["O", "O", "B-LOC"]} {"tokens": ["I", "live", "in", "New", "York"],"tags": ["O", "O", "O", "B-LOC", "I-LOC"]} . . . ``` As you can see, we have two columns in the CSV file. One column is the tokens and the other is the tags. Both the columns are stringified lists! The tokens column contains the tokens of the sentence and the tags column contains the tags for each token. If your CSV is huge, you can divide it into multiple CSV files and upload them separately. Please make sure that the column names are the same in all CSV files. One way to divide the CSV file using pandas is as follows: ```python import pandas as pd # Set the chunk size chunk_size = 1000 i = 1 # Open the CSV file and read it in chunks for chunk in pd.read_csv('example.csv', chunksize=chunk_size): # Save each chunk to a new file chunk.to_csv(f'chunk_{i}.csv', index=False) i += 1 ``` Sample dataset from HuggingFace Hub: [conll2003](https://huggingface.co/datasets/eriktks/conll2003) ## Columns Your CSV/JSONL dataset must have two columns: `tokens` and `tags`. ## Parameters [[autodoc]] trainers.token_classification.params.TokenClassificationParams --- ### Source/Autotrain Api # AutoTrain API With AutoTrain API, you can run your own instance of AutoTrain and use it to train models on Hugging Face Spaces infrastructure (local training coming soon). This API is designed to be used with autotrain compatible models and datasets, and it provides a simple interface to train models with minimal configuration. ## Getting Started To get started with AutoTrain API, all you need to do is install `autotrain-advanced` as discussed in running locally section and run the autotrain app command: ```bash $ autotrain app --port 8000 --host 127.0.0.1 ``` You can then access the API reference at `http://127.0.0.1:8000/docs`. ## Example Usage ```bash curl -X POST "http://127.0.0.1:8000/api/create_project" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer hf_XXXXX" \ -d '{ "username": "abhishek", "project_name": "my-autotrain-api-model", "task": "llm:orpo", "base_model": "meta-llama/Meta-Llama-3-8B-Instruct", "hub_dataset": "argilla/distilabel-capybara-dpo-7k-binarized", "train_split": "train", "hardware": "spaces-a10g-large", "column_mapping": { "text_column": "chosen", "rejected_text_column": "rejected", "prompt_text_column": "prompt" }, "params": { "block_size": 1024, "model_max_length": 4096, "max_prompt_length": 512, "epochs": 1, "batch_size": 2, "lr": 0.00003, "peft": true, "quantization": "int4", "target_modules": "all-linear", "padding": "right", "optimizer": "adamw_torch", "scheduler": "linear", "gradient_accumulation": 4, "mixed_precision": "fp16", "chat_template": "chatml" } }' ``` --- ### Source/Col Map # Understanding Column Mapping Column mapping is a critical setup process in AutoTrain that informs the system about the roles of different columns in your dataset. Whether it's a tabular dataset, text classification data, or another type, the need for precise column mapping ensures that AutoTrain processes each dataset element correctly. ## How Column Mapping Works AutoTrain has no way of knowing what the columns in your dataset represent. AutoTrain requires a clear understanding of each column's function within your dataset to train models effectively. This is managed through a straightforward mapping system in the user interface, represented as a dictionary. Here's a typical example: ``` {"text": "text", "label": "target"} ``` In this example, the `text` column in your dataset corresponds to the text data AutoTrain uses for processing, and the `target` column is treated as the label for training. But let's not get confused! AutoTrain has a way to understand what each column in your dataset represents. If your data is already in AutoTrain format, you dont need to change column mappings. If not, you can easily map the columns in your dataset to the correct AutoTrain format. In the UI, you will see column mapping as a dictionary: ``` {"text": "text", "label": "target"} ``` Here, the column `text` in your dataset is mapped to the AutoTrain column `text`, and the column `target` in your dataset is mapped to the AutoTrain column `label`. Let's say you are training a text classification model and your dataset has the following columns: ``` full_text, target_sentiment "this movie is great", positive "this movie is bad", negative ``` You can map these columns to the AutoTrain format as follows: ``` {"text": "full_text", "label": "target_sentiment"} ``` If your dataset has the columns: `text` and `label`, you don't need to change the column mapping. Let's take a look at column mappings for each task: ## LLM Note: For all LLM tasks, if the text column(s) is not formatted i.e. if contains samples in chat format (dict or json), then you should use `chat_template` parameter. Read more about it in LLM Parameters Section. ### SFT / Generic Trainer ``` {"text": "text"} ``` `text`: The column in your dataset that contains the text data. ### Reward Trainer ``` {"text": "text", "rejected_text": "rejected_text"} ``` `text`: The column in your dataset that contains the text data. `rejected_text`: The column in your dataset that contains the rejected text data. ### DPO / ORPO Trainer ``` {"prompt": "prompt", "text": "text", "rejected_text": "rejected_text"} ``` `prompt`: The column in your dataset that contains the prompt data. `text`: The column in your dataset that contains the text data. `rejected_text`: The column in your dataset that contains the rejected text data. ## Text Classification & Regression, Seq2Seq For text classification and regression, the column mapping should be as follows: ``` {"text": "dataset_text_column", "label": "dataset_target_column"} ``` `text`: The column in your dataset that contains the text data. `label`: The column in your dataset that contains the target variable. ## Token Classification ``` {"text": "tokens", "label": "tags"} ``` `text`: The column in your dataset that contains the tokens. These tokens must be a list of strings. `label`: The column in your dataset that contains the tags. These tags must be a list of strings. For token classification, if you are using a CSV, make sure that the columns are stringified lists. ## Tabular Classification & Regression ``` {"id": "id", "label": ["target"]} ``` `id`: The column in your dataset that contains the unique identifier for each row. `label`: The column in your dataset that contains the target variable. This should be a list of strings. For a single target column, you can pass a list with a single element. For multiple target columns, e.g. a multi label classification task, you can pass a list with multiple elements. # Image Classification For image classification, the column mapping should be as follows: ``` {"image": "image_column", "label": "label_column"} ``` Image classification requires column mapping only when you are using a dataset from Hugging Face Hub. For uploaded datasets, leave column mapping as it is. # Sentence Transformers For all sentence transformers tasks, one needs to map columns to `sentence1_column`, `sentence2_column`, `sentence3_column` & `target_column` column. Not all columns need to be mapped for all trainers of sentence transformers. ## `pair`: ``` {"sentence1_column": "anchor", "sentence2_column": "positive"} ``` ## `pair_class`: ``` {"sentence1_column": "premise", "sentence2_column": "hypothesis", "target_column": "label"} ``` ## `pair_score`: ``` {"sentence1_column": "sentence1", "sentence2_column": "sentence2", "target_column": "score"} ``` ## `triplet`: ``` {"sentence1_column": "anchor", "sentence2_column": "positive", "sentence3_column": "negative"} ``` ## `qa`: ``` {"sentence1_column": "query", "sentence2_column": "answer"} ``` # Extractive Question Answering For extractive question answering, the column mapping should be as follows: ``` {"text": "context", "question": "question", "answer": "answers"} ``` where `answer` is a dictionary with keys `text` and `answer_start`. ## Ensuring Accurate Mapping To ensure your model trains correctly: - Verify Column Names: Double-check that the names used in the mapping dictionary accurately reflect those in your dataset. - Format Appropriately: Especially in token classification, ensure your data format matches expectations (e.g., lists of strings). - Update Mappings for New Datasets: Each new dataset might require its unique mappings based on its structure and the task at hand. By following these guidelines and using the provided examples as templates, you can effectively instruct AutoTrain on how to interpret and handle your data for various machine learning tasks. This process is fundamental for achieving optimal results from your model training endeavors. --- ### Source/Config # AutoTrain Configs AutoTrain Configs are the way to use and train models using AutoTrain locally. Once you have installed AutoTrain Advanced, you can use the following command to train models using AutoTrain config files: ```bash $ export HF_USERNAME=your_hugging_face_username $ export HF_TOKEN=your_hugging_face_write_token $ autotrain --config path/to/config.yaml ``` Example configurations for all tasks can be found in the `configs` directory of the [AutoTrain Advanced GitHub repository](https://github.com/huggingface/autotrain-advanced). Here is an example of an AutoTrain config file: ```yaml task: llm base_model: meta-llama/Meta-Llama-3-8B-Instruct project_name: autotrain-llama3-8b-orpo log: tensorboard backend: local data: path: argilla/distilabel-capybara-dpo-7k-binarized train_split: train valid_split: null chat_template: chatml column_mapping: text_column: chosen rejected_text_column: rejected params: trainer: orpo block_size: 1024 model_max_length: 2048 max_prompt_length: 512 epochs: 3 batch_size: 2 lr: 3e-5 peft: true quantization: int4 target_modules: all-linear padding: right optimizer: adamw_torch scheduler: linear gradient_accumulation: 4 mixed_precision: bf16 hub: username: ${HF_USERNAME} token: ${HF_TOKEN} push_to_hub: true ``` In this config, we are finetuning the `meta-llama/Meta-Llama-3-8B-Instruct` model on the `argilla/distilabel-capybara-dpo-7k-binarized` dataset using the `orpo` trainer for 3 epochs with a batch size of 2 and a learning rate of `3e-5`. More information on the available parameters can be found in the *Data Formats and Parameters* section. In case you dont want to push the model to hub, you can set `push_to_hub` to `false` in the config file. If not pushing the model to hub username and token are not required. Note: they may still be needed if you are trying to access gated models or datasets. --- ### Source/Cost # How much does it cost? AutoTrain offers an accessible approach to model training, providing deployable models with just a few clicks. Understanding the cost involved is essential to planning and executing your projects efficiently. ## Local Usage When you choose to use AutoTrain locally on your own hardware, there is no cost. This option is ideal for those who prefer to manage their own infrastructure and do not require the scalability that cloud resources offer. ## Using AutoTrain on Hugging Face Spaces **Pay-As-You-Go**: Costs for using AutoTrain in Hugging Face Spaces are based on the computing resources you consume. This flexible pricing structure ensures you only pay for what you use, making it cost-effective and scalable for projects of any size. **Ownership and Portability**: Unlike some other platforms, AutoTrain does not retain ownership of your models. Once training is complete, you are free to download and deploy your models wherever you choose, providing flexibility and control over your all your assets. ### Pricing Details **Resource-Based Billing**: Charges are accrued per minute according to the type of hardware utilized during training. This means you can scale your resource usage based on the complexity and needs of your projects. For a detailed breakdown of the costs associated with using Hugging Face Spaces, please refer to the [pricing](https://huggingface.co/pricing#spaces) section on our website. To access the paid features of AutoTrain, you must have a valid payment method on file. You can manage your payment options and view your billing information in the [billing section of your Hugging Face account settings.](https://huggingface.co/settings/billing) By offering both free and flexible paid options, AutoTrain ensures that users can choose the most suitable model training solution for their needs, whether they are experimenting on a local machine or scaling up operations on Hugging Face Spaces. --- ### Source/Faq # Frequently Asked Questions ## Are my data and models secure? Yes, your data and models are secure. AutoTrain uses the Hugging Face Hub to store your data and models. All your data and models are uploaded to your Hugging Face account as private repositories and are only accessible by you. Read more about security [here](https://huggingface.co/docs/hub/en/security). ## Do you upload my data to the Hugging Face Hub? AutoTrain will not upload your dataset to the Hub if you are using the local backend or training in the same space. AutoTrain will push your dataset to the Hub if you are using features like: DGX Cloud or using local CLI to train on Hugging Face's infrastructure. You can safely remove the dataset from the Hub after training is complete. If uploaded, the dataset will be stored in your Hugging Face account as a private repository and will only be accessible by you and the training process. It is not used once the training is complete. ## My training space paused for no reason mid-training AutoTrain Training Spaces will pause itself after training is done (or failed). This is done to save resources and costs. If your training failed, you can still see the space logs and find out what went wrong. Note: you won't be able to retrive the logs if you restart the space. Another reason for the space to pause is if the space is space's sleep time kicking in. If you have a long running training job, you must set the sleep time to a much higher value. The space will anyways pause itself after the training is done thus saving you costs. ## I get error `Your installed package nvidia-ml-py is corrupted. Skip patch functions` This error can be safely ignored. It is a warning from the `nvitop` library and does not affect the functionality of AutoTrain. ## I get 409 conflict error when using the UI This error occurs when you try to create a project with the same name as an existing project. To resolve this error, you can either delete the existing project or create a new project with a different name. This error can also occur when you are trying to train a model while a model is already training in the same space or locally. ## The model I want to use doesn't show up in the model selection dropdown. If the model you want to use is not available in the model selection dropdown, you can add it in the environment variable `AUTOTRAIN_CUSTOM_MODELS` in the space settings. For example, if you want to add the `xxx/yyy` model, go to space settings, create a variable named `AUTOTRAIN_CUSTOM_MODELS` and set the value to `xxx/yyy`. You can also pass the model name as query parameter in the URL. For example, if you want to use the `xxx/yyy` model, you can use the URL `https://huggingface.co/spaces/your_autotrain_space?custom_models=xxx/yyy`. ## How do I use AutoTrain locally? AutoTrain can be used locally by installing the AutoTrain Advanced pypi package. You can read more in *Use AutoTrain Locally* section. ## Can I run AutoTrain on Colab? To start the UI on Colab, you can simply click on the following link: [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain.ipynb) Please note, to run the app on Colab, you will need an ngrok token. You can get one by signing up for free on [ngrok](https://ngrok.com/). This is because Colab does not allow exposing ports to the internet directly. To use the CLI instead on Colab, you can follow the same instructions as for using AutoTrain locally. ## Does AutoTrain have a docker image? Yes, AutoTrain has a docker image. You can find the docker image on Docker Hub [here](https://hub.docker.com/r/huggingface/autotrain-advanced). ## Is windows supported? Unfortunately, AutoTrain does not officially support Windows at the moment. You can try using WSL (Windows Subsystem for Linux) to run AutoTrain on Windows or the docker image. ## "--project-name" argument can not be set as a directory `--project-name` argument should not be a path. it will be created where autotrain command is run. This parameter must be alphanumeric and can contain hypens. ## I am getting `config.json` not found error This means you have trained an adapter model (peft=true) which doesnt generate config.json. It doesnt matter though, the model can still be loaded with AutoModelForCausalLM or with Inference endpoints. If you want to merge weights with base models, you can use `autotrain tools`. Please read about it in miscelleneous section. ## Does autotrain support multi-gpu training? Yes, autotrain supports multi-gpu training. AutoTrain will determine on its own if the user is running the command on a multi-gpu setup and will use multi-gpu ddp if number of gpus is greater than 1 and less than 4 and deepspeed if number of gpus is greater than or equal to 4. ## How can i use a hub dataset with multiple configs? If your hub dataset has multiple configs, you can use `train_split` parameter to specify the both the config and the split. For example, in this dataset [here](https://huggingface.co/datasets/timdettmers/openassistant-guanaco), there are multiple configs: `pair`, `pair-class`, `pair-score` and `triplet`. If i want to use `train` split of `pair-class` config, i can use write `pair-class:train` as `train_split` in the UI or the CLI / config. An example config is shown below: ```yaml data: path: sentence-transformers/all-nli train_split: pair-class:train valid_split: pair-class:test column_mapping: sentence1_column: premise sentence2_column: hypothesis target_column: label ``` --- ### Source/Index # AutoTrain > [!WARNING] > **This project is no longer maintained.** No new features will be added and bugs will not be fixed. We recommend using [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), [TRL](https://github.com/huggingface/trl), or [transformers.Trainer](https://huggingface.co/docs/transformers/main_classes/trainer). 🤗 AutoTrain Advanced (or simply AutoTrain), developed by Hugging Face, is a robust no-code platform designed to simplify the process of training state-of-the-art models across multiple domains: Natural Language Processing (NLP), Computer Vision (CV), and even Tabular Data analysis. This tool leverages the powerful frameworks created by various teams at Hugging Face, making advanced machine learning and artificial intelligence accessible to a broader audience without requiring deep technical expertise. ## Who should use AutoTrain? AutoTrain is the perfect tool for anyone eager to dive into the world of machine learning without getting bogged down by the complexities of model training. Whether you're a business professional, researcher, educator, or hobbyist, AutoTrain offers the simplicity of a no-code interface while still providing the capabilities necessary to develop sophisticated models tailored to your unique datasets. AutoTrain is for anyone who wants to train a state-of-the-art model for a NLP, CV, Speech or even Tabular task, but doesn't want to spend time on the technical details of training a model. Our mission is to democratize machine learning technology, ensuring it is not only accessible to data scientists and ML engineers but also to those without a technical background. If you're looking to harness the power of AI for your projects, AutoTrain is your answer. ## How to use AutoTrain? We offer several ways to use AutoTrain: - No code users can use `AutoTrain Advanced` by creating a new space with AutoTrain Docker image: [Click here](https://huggingface.co/login?next=/spaces/autotrain-projects/autotrain-advanced?duplicate=true) to create AutoTrain Space. Remember to keep your space private and ensure it is equipped with the necessary hardware resources (GPU) for optimal performance. - If you prefer a more hands-on approach, AutoTrain Advanced can also be run locally through its intuitive UI or accessed via the Python API provided in the autotrain-advanced package. This flexibility allows developers to integrate AutoTrain capabilities directly into their projects, customize workflows, and enhance their toolsets with advanced machine learning functionalities. By bridging the gap between cutting-edge technology and practical usability, AutoTrain Advanced empowers users to achieve remarkable results in AI without the need for extensive programming knowledge. Start your journey with AutoTrain today and unlock the potential of machine learning for your projects! ## Walkthroughs To get started with AutoTrain, check out our walkthroughs and tutorials: - [Extractive Question Answering with AutoTrain](https://huggingface.co/blog/abhishek/extractive-qa-autotrain) - [Finetuning PaliGemma with AutoTrain](https://huggingface.co/blog/abhishek/paligemma-finetuning-autotrain) - [Training an Object Detection Model with AutoTrain](https://huggingface.co/blog/abhishek/object-detection-autotrain) - [How to Fine-Tune Custom Embedding Models Using AutoTrain](https://huggingface.co/blog/abhishek/finetune-custom-embeddings-autotrain) - [Train Custom Models on Hugging Face Spaces with AutoTrain SpaceRunner](https://huggingface.co/blog/abhishek/autotrain-spacerunner) - [How to Finetune phi-3 on MacBook Pro](https://huggingface.co/blog/abhishek/phi3-finetune-macbook) - [Finetune Mixtral 8x7B with AutoTrain](https://huggingface.co/blog/abhishek/autotrain-mixtral-dgx-cloud-local) - [Easily Train Models with H100 GPUs on NVIDIA DGX Cloud](https://huggingface.co/blog/train-dgx-cloud) --- ### Source/Quickstart # Quickstart Guide for Local Training This quickstart is for local installation and usage. If you want to use AutoTrain on Hugging Face Spaces, please refer to the *AutoTrain on Hugging Face Spaces* section. You can install AutoTrain Advanced using pip: ```bash $ pip install autotrain-advanced ``` It is advised to install autotrain-advanced in a virtual environment to avoid any conflicts with other packages. Note: AutoTrain doesn't install pytorch, torchaudio, torchvision, or any other large dependencies. You will need to install them separately. ```bash $ conda create -n autotrain python=3.10 $ conda activate autotrain $ pip install autotrain-advanced $ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia $ conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc $ conda install xformers -c xformers $ python -m nltk.downloader punkt $ pip install flash-attn --no-build-isolation # if you want to use flash-attn $ pip install deepspeed # if you want to use deepspeed ```` # Running AutoTrain User Interface (UI) To run the autotrain app locally, you can use the following command: ```bash $ export HF_TOKEN=your_hugging_face_write_token $ autotrain app --host 127.0.0.1 --port 8000 ``` This will start the app on `http://127.0.0.1:8000`. # Using AutoTrain Command Line Interface (CLI) It is also possible to use the CLI: ```bash $ export HF_TOKEN=your_hugging_face_write_token $ autotrain --help ``` This will show the CLI commands that can be used: ```bash usage: autotrain [] positional arguments: { app, llm, setup, api, text-classification, text-regression, image-classification, tabular, spacerunner, seq2seq, token-classification } commands options: -h, --help show this help message and exit --version, -v Display AutoTrain version --config CONFIG Optional configuration file For more information about a command, run: `autotrain --help` ``` It is advised to use only the `autotrain --config CONFIG_FILE` command for training when using the CLI. The autotrain commands that end users will be interested in are: - `app`: Start the AutoTrain UI - `llm`: Train a language model - `text-classification`: Train a text classification model - `text-regression`: Train a text regression model - `image-classification`: Train an image classification model - `tabular`: Train a tabular model - `spacerunner`: Train any custom model using SpaceRunner - `seq2seq`: Train a sequence-to-sequence model - `token-classification`: Train a token classification model Note: above commands are not required if you use preferred `autotrain --config CONFIG_FILE` command to train the models. --- ### Source/Quickstart Py # Quickstart with Python AutoTrain is a library that allows you to train state of the art models on Hugging Face Spaces, or locally. It provides a simple and easy-to-use interface to train models for various tasks like llm finetuning, text classification, image classification, object detection, and more. In this quickstart guide, we will show you how to train a model using AutoTrain in Python. ## Getting Started AutoTrain can be installed using pip: ```bash $ pip install autotrain-advanced ``` The example code below shows how to finetune an LLM model using AutoTrain in Python: ```python import os from autotrain.params import LLMTrainingParams from autotrain.project import AutoTrainProject params = LLMTrainingParams( model="meta-llama/Llama-3.2-1B-Instruct", data_path="HuggingFaceH4/no_robots", chat_template="tokenizer", text_column="messages", train_split="train", trainer="sft", epochs=3, batch_size=1, lr=1e-5, peft=True, quantization="int4", target_modules="all-linear", padding="right", optimizer="paged_adamw_8bit", scheduler="cosine", gradient_accumulation=8, mixed_precision="bf16", merge_adapter=True, project_name="autotrain-llama32-1b-finetune", log="tensorboard", push_to_hub=True, username=os.environ.get("HF_USERNAME"), token=os.environ.get("HF_TOKEN"), ) backend = "local" project = AutoTrainProject(params=params, backend=backend, process=True) project.create() ``` In this example, we are finetuning the `meta-llama/Llama-3.2-1B-Instruct` model on the `HuggingFaceH4/no_robots` dataset. We are training the model for 3 epochs with a batch size of 1 and a learning rate of `1e-5`. We are using the `paged_adamw_8bit` optimizer and the `cosine` scheduler. We are also using mixed precision training with a gradient accumulation of 8. The final model will be pushed to the Hugging Face Hub after training. To train the model, run the following command: ```bash $ export HF_USERNAME= $ export HF_TOKEN= $ python train.py ``` This will create a new project directory with the name `autotrain-llama32-1b-finetune` and start the training process. Once the training is complete, the model will be pushed to the Hugging Face Hub. Your HF_TOKEN and HF_USERNAME are only required if you want to push the model or if you are accessing a gated model or dataset. ## AutoTrainProject Class [[autodoc]] project.AutoTrainProject ## Parameters ### Text Tasks [[autodoc]] trainers.clm.params.LLMTrainingParams [[autodoc]] trainers.sent_transformers.params.SentenceTransformersParams [[autodoc]] trainers.seq2seq.params.Seq2SeqParams [[autodoc]] trainers.token_classification.params.TokenClassificationParams [[autodoc]] trainers.extractive_question_answering.params.ExtractiveQuestionAnsweringParams [[autodoc]] trainers.text_classification.params.TextClassificationParams [[autodoc]] trainers.text_regression.params.TextRegressionParams ### Image Tasks [[autodoc]] trainers.image_classification.params.ImageClassificationParams [[autodoc]] trainers.image_regression.params.ImageRegressionParams [[autodoc]] trainers.object_detection.params.ObjectDetectionParams ### Tabular Tasks [[autodoc]] trainers.tabular.params.TabularParams --- ### Source/Quickstart Spaces # Quickstart Guide to AutoTrain on Hugging Face Spaces AutoTrain on Hugging Face Spaces is the preferred choice for a streamlined experience in model training. This platform is optimized for ease of use, with pre-installed dependencies and managed hardware resources. AutoTrain on Hugging Face Spaces can be used both by no-code users and developers, making it versatile for various levels of expertise. ## Creating a New AutoTrain Space Getting started with AutoTrain is straightforward. Here’s how you can create your new space: 1. **Visit the AutoTrain Page**: To create a new space with AutoTrain Docker image, all you need to do is go to [AutoTrain Homepage](https://hf.co/autotrain) and click on "Create new project". 2. **Log In or View the Setup Screen**: If not logged in, you'll be prompted to do so. Then, you’ll see a screen similar to this: 3. **Set Up Your Space**: - **Choose a Space Name**: Name your space something relevant to your project. - **Allocate Hardware Resources**: Select the necessary computational resources based on your project needs. - **Duplicate Space**: Click on "Duplicate Space" to initiate your AutoTrain space with the Docker image. 4. **Configuration Options**: - PAUSE_ON_FAILURE: Set this to 0 if you prefer the space not to pause on training failures, useful for running continuous experiments. This option can also be used if you continuously want to perfom many experiments in the same space. 5. **Launch and Train**: - Once done, in a few seconds, the AutoTrain Space will be up and running and you will be presented with the following screen: - From here, you can select tasks, upload datasets, choose models, adjust hyperparameters (if needed), and start the training process directly within the space. - The space will manage its own activity, shutting down post-training unless configured otherwise based on the `PAUSE_ON_FAILURE` setting. 6. **Monitoring Progress**: - All training logs and progress can be monitored via TensorBoard, accessible under `username/project_name` on the Hugging Face Hub. - Once training concludes successfully, you’ll find the model files in the same repository. 7. **Navigating the UI**: - If you need help understanding any UI elements, click on the small (i) information icons for detailed descriptions. If you are confused about the UI elements, click on the small (i) information icon to get more information about the UI element. For data formats and detailed parameter information, please see the Data Formats and Parameters section where we provide example datasets and detailed information about the parameters for each task supported by AutoTrain. ## Ensuring Your AutoTrain is Up-to-Date We are constantly adding new features and tasks to AutoTrain Advanced. To benefit from the latest features, tasks, and bug fixes, update your AutoTrain space regularly: - *Factory Reboot*: Navigate to the settings page of your space and click on "Factory reboot" to upgrade to the latest version of AutoTrain Advanced. - *Note*: Simply "restarting" the space does not update it; a factory reboot is necessary for a complete update. For additional details on data formats and specific parameters, refer to the 'Data Formats and Parameters' section where we provide example datasets and extensive parameter information for each supported task by AutoTrain. With these steps, you can effortlessly initiate and manage your AutoTrain projects on Hugging Face Spaces, leveraging the platform's robust capabilities for your machine learning and AI needs. --- ### Source/Support # Help and Support If you need assistance with AutoTrain Advanced or have questions about your projects, you can reach out through several dedicated support channels. We're here to help you navigate any issues you encounter, from technical queries to billing concerns. Below are the best ways to get support: - For technical support or to report a bug, you can [create an issue](https://github.com/huggingface/autotrain-advanced/issues/new) directly in the AutoTrain Advanced GitHub repository. GitHub repo is ideal for tracking bugs, requesting features, or getting help with troubleshooting problems. When submitting an issue, please include all the details in question to help us provide the most relevant support quickly. - [Ask in the Hugging Face Forum](https://discuss.huggingface.co/c/autotrain/16). This space is perfect for asking questions, sharing your experiences, or discussing AutoTrain with other users and the Hugging Face team. The forum is a great resource for getting advice, learning best practices, and connecting with other machine learning practitioners. - For enterprise users or specific inquiries related to billing, please [email us](mailto:autotrain@hf.co) directly. This channel ensures that your more sensitive or account-specific issues are handled appropriately and confidentially. When emailing, please provide your username and project name so we can assist you efficiently. Please note: e-mail support is only available for pro/enterprise users or those with specific queries about billing. By utilizing these support channels, you can ensure that any hurdles you face while using AutoTrain Advanced are addressed promptly, allowing you to focus on achieving your project goals. Whether you're a beginner or an experienced user, we are here to support your journey in AI model training. --- ### README # 🤗 AutoTrain Advanced > [!WARNING] > **This project is no longer maintained.** No new features will be added and bugs will not be fixed. We recommend using [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), [TRL](https://github.com/huggingface/trl), or [transformers.Trainer](https://huggingface.co/docs/transformers/main_classes/trainer). AutoTrain Advanced: faster and easier training and deployments of state-of-the-art machine learning models. AutoTrain Advanced is a no-code solution that allows you to train machine learning models in just a few clicks. Please note that you must upload data in correct format for project to be created. For help regarding proper data format and pricing, check out the documentation. NOTE: AutoTrain is free! You only pay for the resources you use in case you decide to run AutoTrain on Hugging Face Spaces. When running locally, you only pay for the resources you use on your own infrastructure. ## Supported Tasks | Task | Status | Python Notebook | Example Configs | | --- | --- | --- | --- | | LLM SFT Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | [llm_sft_finetune.yaml](https://github.com/huggingface/autotrain-advanced/blob/main/configs/llm_finetuning/smollm2.yml) | | LLM ORPO Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | [llm_orpo_finetune.yaml](https://github.com/huggingface/autotrain-advanced/blob/main/configs/llm_finetuning/llama3-8b-orpo.yml) | | LLM DPO Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | [llm_dpo_finetune.yaml](https://github.com/huggingface/autotrain-advanced/blob/main/configs/llm_finetuning/llama3-8b-dpo-qlora.yml) | | LLM Reward Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | [llm_reward_finetune.yaml](https://github.com/huggingface/autotrain-advanced/blob/main/configs/llm_finetuning/llama32-1b-sft.yml) | | LLM Generic/Default Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | [llm_generic_finetune.yaml](https://github.com/huggingface/autotrain-advanced/blob/main/configs/llm_finetuning/gpt2_sft.yml) | | Text Classification | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/text_classification.ipynb) | [text_classification.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/text_classification) | | Text Regression | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/text_regression.ipynb) | [text_regression.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/text_regression) | | Token Classification | ✅ | Coming Soon | [token_classification.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/token_classification) | | Seq2Seq | ✅ | Coming Soon | [seq2seq.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/seq2seq) | | Extractive Question Answering | ✅ | Coming Soon | [extractive_qa.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/extractive_question_answering) | | Image Classification | ✅ | Coming Soon | [image_classification.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/image_classification) | | Image Scoring/Regression | ✅ | Coming Soon | [image_regression.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/image_scoring) | | VLM | 🟥 | Coming Soon | [vlm.yaml](https://github.com/huggingface/autotrain-advanced/tree/main/configs/vlm) | ## Running UI on Colab or Hugging Face Spaces - Deploy AutoTrain on Hugging Face Spaces: [](https://huggingface.co/login?next=%2Fspaces%2Fautotrain-projects%2Fautotrain-advanced%3Fduplicate%3Dtrue) - Run AutoTrain UI on Colab via ngrok: [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_ngrok.ipynb) ## Local Installation You can Install AutoTrain-Advanced python package via PIP. Please note you will need python >= 3.10 for AutoTrain Advanced to work properly. pip install autotrain-advanced Please make sure that you have git lfs installed. Check out the instructions here: https://github.com/git-lfs/git-lfs/wiki/Installation You also need to install torch, torchaudio and torchvision. The best way to run autotrain is in a conda environment. You can create a new conda environment with the following command: conda create -n autotrain python=3.10 conda activate autotrain pip install autotrain-advanced conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc Once done, you can start the application using: autotrain app --port 8080 --host 127.0.0.1 If you are not fond of UI, you can use AutoTrain Configs to train using command line or simply AutoTrain CLI. To use config file for training, you can use the following command: autotrain --config You can find sample config files in the `configs` directory of this repository. Example config file for finetuning SmolLM2: ```yaml task: llm-sft base_model: HuggingFaceTB/SmolLM2-1.7B-Instruct project_name: autotrain-smollm2-finetune log: tensorboard backend: local data: path: HuggingFaceH4/no_robots train_split: train valid_split: null chat_template: tokenizer column_mapping: text_column: messages params: block_size: 2048 model_max_length: 4096 epochs: 2 batch_size: 1 lr: 1e-5 peft: true quantization: int4 target_modules: all-linear padding: right optimizer: paged_adamw_8bit scheduler: linear gradient_accumulation: 8 mixed_precision: bf16 merge_adapter: true hub: username: ${HF_USERNAME} token: ${HF_TOKEN} push_to_hub: true ``` To fine-tune a model using the config file above, you can use the following command: ```bash $ export HF_USERNAME= $ export HF_TOKEN= $ autotrain --config ``` ## Documentation Documentation is available at https://hf.co/docs/autotrain/ ## Citation ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ---