README Copy Section

Generating the documentation

To generate the documentation, you have to build it. Several packages are necessary to build the doc.

First, you need to install the project itself by running the following command at the root of the code repository:

pip install -e .

You also need to install 2 extra packages:

hf-doc-builder to build the docs

pip install git+https://github.com/huggingface/doc-builder@main
watchdog for live reloads

pip install watchdog

---
NOTE

You only need to generate the documentation to inspect it locally (if you're planning changes and want to
check how they look before committing for instance). You don't have to commit the built documentation.

---

Building the documentation Copy Chapter

Once you have setup the doc-builder and additional packages with the pip install command above,
you can generate the documentation by typing the following command:

doc-builder build autotrain docs/source/ --build_dir ~/tmp/test-build

You can adapt the --build_dir to set any temporary folder that you prefer. This command will create it and generate
the MDX files that will be rendered as the documentation on the main website. You can inspect them in your favorite
Markdown editor.

Previewing the documentation Copy Chapter

To preview the docs, run the following command:

doc-builder preview autotrain docs/source/

The docs will be viewable at http://localhost:5173. You can also preview the docs once you
have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives.

---
NOTE

The preview command only works with existing doc files. When you add a completely new file, you need to update
_toctree.yml & restart preview command (ctrl-c to stop it & call doc-builder preview ... again).

---

---

Source/Tasks/Extractive Qa Copy Section

Extractive Question Answering with AutoTrain

Extractive Question Answering (QA) enables AI models to find and extract precise answers from text passages. This guide shows you how to train custom QA models using AutoTrain, supporting popular architectures like BERT, RoBERTa, and DeBERTa.

What is Extractive Question Answering? Copy Chapter

Extractive QA models learn to:
- Locate exact answer spans within longer text passages
- Understand questions and match them to relevant context
- Extract precise answers rather than generating them
- Handle both simple and complex queries about the text

Preparing your Data Copy Chapter

Your dataset needs these essential columns:
- text: The passage containing potential answers (also called context)
- question: The query you want to answer
- answer: Answer span information including text and position

Here is an example of how your dataset should look:

/ Detailed source-code truncated for AI context efficiency. /

Note: the preferred format for question answering is JSONL, if you want to use CSV, the answer column should be stringified JSON with the keys text and answer_start.

Example dataset from Hugging Face Hub: lhoestq/squad

P.S. You can use both squad and squad v2 data format with correct column mappings.

Training Options Copy Chapter

Local Training Copy Section

To train an Extractive QA model locally, you need a config file:

task: extractive-qa
base_model: google-bert/bert-base-uncased
project_name: autotrain-bert-ex-qa1
log: tensorboard
backend: local
data:
  path: lhoestq/squad
  train_split: train
  valid_split: validation
  column_mapping:
    text_column: context
    question_column: question
    answer_column: answers
params:
  max_seq_length: 512
  max_doc_stride: 128
  epochs: 3
  batch_size: 4
  lr: 2e-5
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 1
  mixed_precision: fp16
hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

To train the model, run the following command:

$ autotrain --config config.yaml

Here, we are training a BERT model on the SQuAD dataset using the Extractive QA task. The model is trained for 3 epochs with a batch size of 4 and a learning rate of 2e-5. The training process is logged using TensorBoard. The model is trained locally and pushed to the Hugging Face Hub after training.

Cloud Training on Hugging Face Copy Section

As always, pay special attention to column mapping.

Parameter Reference Copy Chapter

[[autodoc]] trainers.extractive_question_answering.params.ExtractiveQuestionAnsweringParams

---

Source/Tasks/Image Classification Regression Copy Section

Image Classification & Regression

Image classification is a form of supervised learning where a model is trained to identify
and categorize objects within images. AutoTrain simplifies the process, enabling you to
train a state-of-the-art image classification model by simply uploading labeled example
images.

Image regression/scoring is a form of supervised learning where a model is trained to predict a
score or value for an image. AutoTrain simplifies the process, enabling you to train a
state-of-the-art image scoring model by simply uploading labeled example images.

Preparing your data Copy Chapter

To ensure your image classification model trains effectively, follow these guidelines for preparing your data:

Organizing Images For Image Classification Copy Section

Prepare a zip file containing your categorized images. Each category should have its own
subfolder named after the class it represents. For example, to differentiate between
'cats' and 'dogs', your zip file structure should resemble the following:

cats_and_dogs.zip
├── cats
│   ├── cat.1.jpg
│   ├── cat.2.jpg
│   ├── cat.3.jpg
│   └── ...
└── dogs
    ├── dog.1.jpg
    ├── dog.2.jpg
    ├── dog.3.jpg
    └── ...

You can also use a dataset from the Hugging Face Hub. Example dataset from Hugging Face Hub: truepositive/hotdog_nothotdog.

Organizing Images for Image Regression/Scoring Copy Section

Prepare a zip file containing your images and metadata.jsonl.

Archive.zip
├── 0001.png
├── 0002.png
├── 0003.png
├── .
├── .
├── .
└── metadata.jsonl

Example for metadata.jsonl:

{"file_name": "0001.png", "target": 0.5}
{"file_name": "0002.png", "target": 0.7}
{"file_name": "0003.png", "target": 0.3}

Please note that metadata.jsonl should contain the file_name and the target value for each image.

You can also use a dataset from the Hugging Face Hub. Example dataset from Hugging Face Hub: abhishek/img-quality-full.

Image Requirements Copy Section

- Format: Ensure all images are in JPEG, JPG, or PNG format.

- Quantity: Include at least 5 images per class to provide the model with sufficient examples for learning.

- Exclusivity: The zip file should exclusively contain folders named after the classes,
and these folders should only contain relevant images. No additional files or nested
folders should be included.

Additional Tips

- Uniformity: While not required, having images of similar sizes and resolutions can help improve model performance.

- Variability: Include a variety of images for each class to encompass the range of
appearances and contexts the model might encounter in real-world scenarios.

Some points to keep in mind:

- The zip file should contain multiple folders (the classes), each folder should contain images of a single class.
- The name of the folder should be the name of the class.
- The images must be jpeg, jpg or png.
- There should be at least 5 images per class.
- There must not be any other files in the zip file.
- There must not be any other folders inside the zip folder.

When train.zip is decompressed, it creates two folders: cats and dogs. these are the two categories for classification. The images for both categories are in their respective folders. You can have as many categories as you want.

Column Mapping Copy Chapter

For image classification, if you are using a zip dataset format, the column mapping should be default and should not be changed.

data:
  .
  .
  .
  column_mapping:
    image_column: image
    target_column: label

For image regression, the column mapping must be as follows:

data:
  .
  .
  .
  column_mapping:
    image_column: image
    target_column: target

For image regression, metadata.jsonl should contain the file_name and the target value for each image.

If you are using a dataset from the Hugging Face Hub, you should set appropriate column mappings based on the dataset.

Training Copy Chapter

Local Training Copy Section

To train the model locally, create a configuration file (config.yaml) with the following content:

task: image_classification
base_model: google/vit-base-patch16-224
project_name: autotrain-cats-vs-dogs-finetuned
log: tensorboard
backend: local
data:
  path: cats_vs_dogs
  train_split: train
  valid_split: null
  column_mapping:
    image_column: image
    target_column: label
params:
  epochs: 2
  batch_size: 4
  lr: 2e-5
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 1
  mixed_precision: fp16
hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

Here, we are using cats_and_dogs dataset from Hugging Face Hub. The model is trained for 2 epochs with a batch size of 4 and a learning rate of 2e-5. We are using the adamw_torch optimizer and the linear scheduler. We are also using mixed precision training with a gradient accumulation of 1.

In order to use a local dataset, you can change the data section to:

data:
  path: data/
  train_split: train # this folder inside data/ will be used for training, it contains the images in subfolders.
  valid_split: valid # this folder inside data/ will be used for validation, it contains the images in subfolders. can also be null.
  column_mapping:
    image_column: image
    target_column: label

Similarly, for image regression, you can use the following configuration file:

task: image_regression
base_model: microsoft/resnet-50
project_name: autotrain-img-quality-resnet50
log: tensorboard
backend: local
data:
  path: abhishek/img-quality-full
  train_split: train
  valid_split: null
  column_mapping:
    image_column: image
    target_column: target
params:
  epochs: 10
  batch_size: 8
  lr: 2e-3
  optimizer: adamw_torch
  scheduler: cosine
  gradient_accumulation: 1
  mixed_precision: fp16
hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

To train the model, run the following command:

$ autotrain --config config.yaml

This will start the training process and save the model to the Hugging Face Hub after training is complete. In case you dont want to save the model to the hub, you can set push_to_hub to false in the configuration file.

Training on Hugging Face Spaces Copy Section

To train the model on Hugging Face Spaces, create a training space as described in Quickstart section.

An example UI for training an image scoring model on Hugging Face Spaces is shown below:

In this example, we are training an image scoring model using the microsoft/resnet-50 model on the abhishek/img-quality-full dataset.
We are training the model for 3 epochs with a batch size of 8 and a learning rate of 5e-5.
We are using the adamw_torch optimizer and the linear scheduler.
We are also using mixed precision training with a gradient accumulation of 1.

Note how the column mapping has now been changed and target points to quality_mos column in the dataset.

To train the model, click on the Start Training button. This will start the training process and save the model to the Hugging Face Hub after training is complete.

Parameters Copy Chapter

Image Classification Parameters Copy Section

[[autodoc]] trainers.image_classification.params.ImageClassificationParams

Image Regression Parameters Copy Section

[[autodoc]] trainers.image_regression.params.ImageRegressionParams

---

Source/Tasks/Llm Finetuning Copy Section

LLM Finetuning with AutoTrain Advanced

AutoTrain Advanced makes it easy to fine-tune large language models (LLMs) for your specific use cases. This guide covers everything you need to know about LLM fine-tuning.

Key Features Copy Chapter

Supported Training Methods Copy Chapter

Data Preparation Copy Chapter

LLM finetuning accepts data in CSV and JSONL formats. JSONL is the preferred format.
How data is formatted depends on the task you are training the LLM for.

Classic Text Generation Copy Section

For text generation, the data should be in the following format:

| text |
|---------------------------------------------------------------|
| wikipedia is a free online encyclopedia |
| it is a collaborative project |
| that anyone can edit |
| wikipedia is the largest and most popular general reference work on the internet |

An example dataset for this format can be found here: stas/openwebtext-10k

Example tasks:
- Text generation
- Code completion

Compatible trainers:
- SFT Trainer
- Generic Trainer

Chatbot / question-answering / code generation / function calling Copy Section

For this task, you can use CSV or JSONL data. If you are formatting the data yourself (adding start, end tokens, etc.), you can use CSV or JSONL format.
If you do not want to format the data yourself and want --chat-template parameter to format the data for you, you must use JSONL format.
In both cases, CSV and JSONL can be used interchangeably but JSONL is the most preferred format.

To train a chatbot, your data will have content and role. Some models support system role as well.

Here is an example of a chatbot dataset (single sample):

[{'content': 'Help write a letter of 100 -200 words to my future self for '
             'Kyra, reflecting on her goals and aspirations.',
  'role': 'user'},
 {'content': 'Dear Future Self,\n'
             '\n'
             "I hope you're happy and proud of what you've achieved. As I "
             "write this, I'm excited to think about our goals and how far "
             "you've come. One goal was to be a machine learning engineer. I "
             "hope you've worked hard and become skilled in this field. Keep "
             'learning and innovating. Traveling was important to us. I hope '
             "you've seen different places and enjoyed the beauty of our "
             'world. Remember the memories and lessons. Starting a family '
             'mattered to us. If you have kids, treasure every moment. Be '
             'patient, loving, and grateful for your family.\n'
             '\n'
             'Take care of yourself. Rest, reflect, and cherish the time you '
             'spend with loved ones. Remember your dreams and celebrate what '
             "you've achieved. Your determination brought you here. I'm "
             "excited to see the person you've become, the impact you've made, "
             'and the love and joy in your life. Embrace opportunities and '
             'keep dreaming big.\n'
             '\n'
             'With love,\n'
             'Kyra',
  'role': 'assistant'}]

As you can see, the data has content and role columns. The role column can be user or assistant or system.
This data is, however, not formatted for training. You can use the --chat-template parameter to format the data during training.

--chat-template supports the following kinds of templates:
- none (default)
- zephyr
- chatml
- tokenizer: use chat template mentioned in tokenizer config

A multi-line sample is also shown below:

[{"content": "hello", "role": "user"}, {"content": "hi nice to meet you", "role": "assistant"}]
[{"content": "how are you", "role": "user"}, {"content": "I am fine", "role": "assistant"}]
[{"content": "What is your name?", "role": "user"}, {"content": "My name is Mary", "role": "assistant"}]
[{"content": "Which is the best programming language?", "role": "user"}, {"content": "Python", "role": "assistant"}]
.
.
.

An example dataset for this format can be found here: HuggingFaceH4/no_robots

If you dont want to format the data using --chat-template, you can format the data yourself and use the following format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHelp write a letter of 100 -200 words to my future self for Kyra, reflecting on her goals and aspirations.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nDear Future Self,\n\nI hope you're happy and proud of what you've achieved. As I write this, I'm excited to think about our goals and how far you've come. One goal was to be a machine learning engineer. I hope you've worked hard and become skilled in this field. Keep learning and innovating. Traveling was important to us. I hope you've seen different places and enjoyed the beauty of our world. Remember the memories and lessons. Starting a family mattered to us. If you have kids, treasure every moment. Be patient, loving, and grateful for your family.\n\nTake care of yourself. Rest, reflect, and cherish the time you spend with loved ones. Remember your dreams and celebrate what you've achieved. Your determination brought you here. I'm excited to see the person you've become, the impact you've made, and the love and joy in your life. Embrace opportunities and keep dreaming big.\n\nWith love,\nKyra<|eot_id|>

A sample multi-line dataset is shown below:

[{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nhello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nhi nice to meet you<|eot_id|>"}]
[{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nhow are you<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nI am fine<|eot_id|>"}]
[{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is your name?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nMy name is Mary<|eot_id|>"}]
[{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhich is the best programming language?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nPython<|eot_id|>"}]
.
.
.

An example dataset for this format can be found here: timdettmers/openassistant-guanaco

In the examples above, we have seen only two turns: one from the user and one from the assistant. However, you can have multiple turns from the user and assistant in a single sample.

Chat models can be trained using the following trainers:

- SFT Trainer:
- requires only text column
- example dataset: HuggingFaceH4/no_robots

- Generic Trainer:
- requires only text column
- example dataset: HuggingFaceH4/no_robots

- Reward Trainer:
- requires text and rejected_text columns
- example dataset: trl-lib/ultrafeedback_binarized

- DPO Trainer:
- requires prompt, text, and rejected_text columns
- example dataset: trl-lib/ultrafeedback_binarized

- ORPO Trainer:
- requires prompt, text, and rejected_text columns
- example dataset: trl-lib/ultrafeedback_binarized

The only difference between the data format for reward trainer and DPO/ORPO trainer is that the reward trainer requires only text and rejected_text columns, while the DPO/ORPO trainer requires an additional prompt column.

Best Practices for LLM Fine-tuning Copy Chapter

Memory Optimization Copy Section

Data Quality Copy Section

Training Tips Copy Section

Related Resources Copy Section

Training Copy Chapter

Local Training Copy Section

Locally the training can be performed by using autotrain --config config.yaml command. The config.yaml file should contain the following parameters:

task: llm-orpo
base_model: meta-llama/Meta-Llama-3-8B-Instruct
project_name: autotrain-llama3-8b-orpo
log: tensorboard
backend: local
data:
  path: argilla/distilabel-capybara-dpo-7k-binarized
  train_split: train
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected
    prompt_text_column: prompt
params:
  block_size: 1024
  model_max_length: 8192
  max_prompt_length: 512
  epochs: 3
  batch_size: 2
  lr: 3e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 4
  mixed_precision: fp16
hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

In the above config file, we are training a model using the ORPO trainer.
The model is trained on the meta-llama/Meta-Llama-3-8B-Instruct model.
The data is argilla/distilabel-capybara-dpo-7k-binarized dataset. The chat_template parameter is set to chatml.
The column_mapping parameter is used to map the columns in the dataset to the required columns for the ORPO trainer.
The params section contains the training parameters such as block_size, model_max_length, epochs, batch_size, lr, peft, quantization, target_modules, padding, optimizer, scheduler, gradient_accumulation, and mixed_precision.
The hub section contains the username and token for the Hugging Face account and the push_to_hub parameter is set to true to push the trained model to the Hugging Face Hub.

If you have training file locally, you can change data part to:

data:
  path: path/to/training/file
  train_split: train # name of the training file
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected
    prompt_text_column: prompt

The above assumes you have train.csv or train.jsonl in the path/to/training/file directory and you will be applying chatml template to the data.

You can run the training using the following command:

$ autotrain --config config.yaml

More example config files for finetuning different types of lllm and different tasks can be found in the here.

Training in Hugging Face Spaces Copy Section

If you are training in Hugging Face Spaces, everything is the same as local training:

In the UI, you need to make sure you select the right model, the dataset and the splits. Special care should be taken for column_mapping.

Once you are happy with the parameters, you can click on the Start Training button to start the training process.

Parameters Copy Chapter

LLM Fine Tuning Parameters Copy Section

[[autodoc]] trainers.clm.params.LLMTrainingParams

Task specific parameters Copy Section

The length parameters used for different trainers can be different. Some require more context than others.

- block_size: This is the maximum sequence length or length of one block of text. Setting to -1 determines block size automatically. Default is -1.
- model_max_length: Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage. Default is 1024
- max_prompt_length: Specify the maximum length for prompts used in training, particularly relevant for tasks requiring initial contextual input. Used only for orpo and dpo trainer.
- max_completion_length: Completion length to use, for orpo: encoder-decoder models only. For dpo, it is the length of the completion text.

NOTE:
- block size cannot be greater than model_max_length!
- max_prompt_length cannot be greater than model_max_length!
- max_prompt_length cannot be greater than block_size!
- max_completion_length cannot be greater than model_max_length!
- max_completion_length cannot be greater than block_size!

NOTE: Not following these constraints will result in an error / nan losses.

#### Generic Trainer

--add_eos_token, --add-eos-token
                    Toggle whether to automatically add an End Of Sentence (EOS) token at the end of texts, which can be critical for certain
                    types of models like language models. Only used for default trainer
--block_size BLOCK_SIZE, --block-size BLOCK_SIZE
                    Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to
                    -1 determines block size automatically. Default is -1.
--model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH
                    Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage.
                    Default is 1024

#### SFT Trainer

--block_size BLOCK_SIZE, --block-size BLOCK_SIZE
                    Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to
                    -1 determines block size automatically. Default is -1.
--model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH
                    Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage.
                    Default is 1024

#### Reward Trainer

--block_size BLOCK_SIZE, --block-size BLOCK_SIZE
                    Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to
                    -1 determines block size automatically. Default is -1.
--model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH
                    Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage.
                    Default is 1024

#### DPO Trainer

--dpo-beta DPO_BETA, --dpo-beta DPO_BETA
                    Beta for DPO trainer
--model-ref MODEL_REF
                    Reference model to use for DPO when not using PEFT
--block_size BLOCK_SIZE, --block-size BLOCK_SIZE
                    Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to
                    -1 determines block size automatically. Default is -1.
--model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH
                    Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage.
                    Default is 1024
--max_prompt_length MAX_PROMPT_LENGTH, --max-prompt-length MAX_PROMPT_LENGTH
                    Specify the maximum length for prompts used in training, particularly relevant for tasks requiring initial contextual input.
                    Used only for orpo trainer.
--max_completion_length MAX_COMPLETION_LENGTH, --max-completion-length MAX_COMPLETION_LENGTH
                    Completion length to use, for orpo: encoder-decoder models only

#### ORPO Trainer

--block_size BLOCK_SIZE, --block-size BLOCK_SIZE
                    Specify the block size for processing sequences. This is maximum sequence length or length of one block of text. Setting to
                    -1 determines block size automatically. Default is -1.
--model_max_length MODEL_MAX_LENGTH, --model-max-length MODEL_MAX_LENGTH
                    Set the maximum length for the model to process in a single batch, which can affect both performance and memory usage.
                    Default is 1024
--max_prompt_length MAX_PROMPT_LENGTH, --max-prompt-length MAX_PROMPT_LENGTH
                    Specify the maximum length for prompts used in training, particularly relevant for tasks requiring initial contextual input.
                    Used only for orpo trainer.
--max_completion_length MAX_COMPLETION_LENGTH, --max-completion-length MAX_COMPLETION_LENGTH
                    Completion length to use, for orpo: encoder-decoder models only

---

Source/Tasks/Object Detection Copy Section

Object Detection

Object detection is a form of supervised learning where a model is trained to identify
and categorize objects within images. AutoTrain simplifies the process, enabling you to
train a state-of-the-art object detection model by simply uploading labeled example images.

Preparing your data Copy Chapter

To ensure your object detection model trains effectively, follow these guidelines for preparing your data:

Organizing Images Copy Section

Prepare a zip file containing your images and metadata.jsonl.

Archive.zip
├── 0001.png
├── 0002.png
├── 0003.png
├── .
├── .
├── .
└── metadata.jsonl

Example for metadata.jsonl:

{"file_name": "0001.png", "objects": {"bbox": [[302.0, 109.0, 73.0, 52.0]], "category": [0]}}
{"file_name": "0002.png", "objects": {"bbox": [[810.0, 100.0, 57.0, 28.0]], "category": [1]}}
{"file_name": "0003.png", "objects": {"bbox": [[160.0, 31.0, 248.0, 616.0], [741.0, 68.0, 202.0, 401.0]], "category": [2, 2]}}

Please note that bboxes need to be in COCO format [x, y, width, height].

Image Requirements Copy Section

- Format: Ensure all images are in JPEG, JPG, or PNG format.

- Quantity: Include at least 5 images to provide the model with sufficient examples for learning.

- Exclusivity: The zip file should exclusively contain images and metadata.jsonl.
No additional files or nested folders should be included.

Some points to keep in mind:

- The images must be jpeg, jpg or png.
- There should be at least 5 images per split.
- There must not be any other files in the zip file.
- There must not be any other folders inside the zip folder.

When train.zip is decompressed, it creates no folders: only images and metadata.jsonl.

Parameters Copy Chapter

[[autodoc]] trainers.object_detection.params.ObjectDetectionParams

---

Source/Tasks/Sentence Transformer Copy Section

Sentence Transformers

This task lets you easily train or fine-tune a Sentence Transformer model on your own dataset.

AutoTrain supports the following types of sentence transformer finetuning:

- pair: dataset with two sentences: anchor and positive
- pair_class: dataset with two sentences: premise and hypothesis and a target label
- pair_score: dataset with two sentences: sentence1 and sentence2 and a target score
- triplet: dataset with three sentences: anchor, positive and negative
- qa: dataset with two sentences: query and answer

Data Format Copy Chapter

Sentence Transformers finetuning accepts data in CSV/JSONL format. You can also use a dataset from Hugging Face Hub.

pair pair')" title="Copy section prompt for LLMs"> Copy Section

For pair training, the data should be in the following format:

| anchor | positive |
|--------|----------|
| hello | hi |
| how are you | I am fine |
| What is your name? | My name is Abhishek |
| Which is the best programming language? | Python |

pair_class pair_class')" title="Copy section prompt for LLMs"> Copy Section

For pair_class training, the data should be in the following format:

| premise | hypothesis | label |
|---------|------------|-------|
| hello | hi | 1 |
| how are you | I am fine | 0 |
| What is your name? | My name is Abhishek | 1 |
| Which is the best programming language? | Python | 1 |

pair_score pair_score')" title="Copy section prompt for LLMs"> Copy Section

For pair_score training, the data should be in the following format:

| sentence1 | sentence2 | score |
|-----------|-----------|-------|
| hello | hi | 0.8 |
| how are you | I am fine | 0.2 |
| What is your name? | My name is Abhishek | 0.9 |
| Which is the best programming language? | Python | 0.7 |

triplet triplet')" title="Copy section prompt for LLMs"> Copy Section

For triplet training, the data should be in the following format:

| anchor | positive | negative |
|--------|----------|----------|
| hello | hi | bye |
| how are you | I am fine | I am not fine |
| What is your name? | My name is Abhishek | Whats it to you? |
| Which is the best programming language? | Python | Javascript |

qa qa')" title="Copy section prompt for LLMs"> Copy Section

For qa training, the data should be in the following format:

| query | answer |
|-------|--------|
| hello | hi |
| how are you | I am fine |
| What is your name? | My name is Abhishek |
| Which is the best programming language? | Python |

Parameters Copy Chapter

---

Source/Tasks/Seq2seq Copy Section

Seq2Seq

Seq2Seq is a task that involves converting a sequence of words into another sequence of words.
It is used in machine translation, text summarization, and question answering.

Data Format Copy Chapter

You can have the dataset as a CSV file:

text,target
"this movie is great","dieser Film ist großartig"
"this movie is bad","dieser Film ist schlecht"
.
.
.

Or as a JSONL file:

{"text": "this movie is great", "target": "dieser Film ist großartig"}
{"text": "this movie is bad", "target": "dieser Film ist schlecht"}
.
.
.

Columns Copy Chapter

Your CSV/JSONL dataset must have two columns: text and target.

Parameters Copy Chapter

[[autodoc]] trainers.seq2seq.params.Seq2SeqParams

---

Source/Tasks/Tabular Copy Section

Tabular Classification / Regression

Using AutoTrain, you can train a model to classify or regress tabular data easily.
All you need to do is select from a list of models and upload your dataset.
Parameter tuning is done automatically.

Models Copy Chapter

The following models are available for tabular classification / regression.

- xgboost
- random_forest
- ridge
- logistic_regression
- svm
- extra_trees
- gradient_boosting
- adaboost
- decision_tree
- knn

Data Format Copy Chapter

id,category1,category2,feature1,target
1,A,X,0.3373961604172684,1
2,B,Z,0.6481718720511972,0
3,A,Y,0.36824153984054797,1
4,B,Z,0.9571551589530464,1
5,B,Z,0.14035078041264515,1
6,C,X,0.8700872583584364,1
7,A,Y,0.4736080452737105,0
8,C,Y,0.8009107519796442,1
9,A,Y,0.5204774795512048,0
10,A,Y,0.6788795301189603,0
.
.
.

Columns Copy Chapter

Your CSV dataset must have two columns: id and target.

Parameters Copy Chapter

[[autodoc]] trainers.tabular.params.TabularParams

---

Source/Tasks/Text Classification Regression Copy Section

Text Classification & Regression

Training a text classification/regression model with AutoTrain is super-easy! Get your data ready in
proper format and then with just a few clicks, your state-of-the-art model will be ready to
be used in production.

Config file task names:
- text_classification
- text-classification
- text_regression
- text-regression

Data Format Copy Chapter

Text classification/regression supports datasets in both CSV and JSONL formats.

CSV Format Copy Section

Let's train a model for classifying the sentiment of a movie review. The data should be
in the following CSV format:

text,target
"this movie is great",positive
"this movie is bad",negative
.
.
.

As you can see, we have two columns in the CSV file. One column is the text and the other
is the label. The label can be any string. In this example, we have two labels: positive
and negative. You can have as many labels as you want.

And if you would like to train a model for scoring a movie review on a scale of 1-5. The data can be as follows:

text,target
"this movie is great",4.9
"this movie is bad",1.5
.
.
.

JSONL Format Copy Section

{"text": "this movie is great", "target": "positive"}
{"text": "this movie is bad", "target": "negative"}
.
.
.

and for regression:

{"text": "this movie is great", "target": 4.9}
{"text": "this movie is bad", "target": 1.5}
.
.

Column Mapping / Names Copy Section

Your CSV dataset must have two columns: text and target.
If your column names are different than text and target, you can map the dataset column to AutoTrain column names.

Training Copy Chapter

Local Training Copy Section

To train a text classification/regression model locally, you can use the autotrain --config config.yaml command.

Here is an example of a config.yaml file for training a text classification model:

task: text_classification # or text_regression
base_model: google-bert/bert-base-uncased
project_name: autotrain-bert-imdb-finetuned
log: tensorboard
backend: local
data:
  path: stanfordnlp/imdb
  train_split: train
  valid_split: test
  column_mapping:
    text_column: text
    target_column: label
params:
  max_seq_length: 512
  epochs: 3
  batch_size: 4
  lr: 2e-5
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 1
  mixed_precision: fp16
hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

In this example, we are training a text classification model using the google-bert/bert-base-uncased model on the IMDB dataset.
We are using the stanfordnlp/imdb dataset, which is already available on Hugging Face Hub.
We are training the model for 3 epochs with a batch size of 4 and a learning rate of 2e-5.
We are using the adamw_torch optimizer and the linear scheduler.
We are also using mixed precision training with a gradient accumulation of 1.

If you want to use a local CSV/JSONL dataset, you can change the data section to:

data:
  path: data/ # this must be the path to the directory containing the train and valid files
  train_split: train # this must be either train.csv or train.json
  valid_split: valid # this must be either valid.csv or valid.json
  column_mapping:
    text_column: text # this must be the name of the column containing the text
    target_column: label # this must be the name of the column containing the target

To train the model, run the following command:

$ autotrain --config config.yaml

You can find example config files for text classification and regression in the here and here respectively.

Training on Hugging Face Spaces Copy Section

The parameters for training on Hugging Face Spaces are the same as for local training.
If you are using your own dataset, select "Local" as dataset source and upload your dataset.
In the following screenshot, we are training a text classification model using the google-bert/bert-base-uncased model on the IMDB dataset.

For text regression, all you need to do is select "Text Regression" as the task and everything else remains the same (except the data, of course).

Training Parameters Copy Chapter

Training parameters for text classification and regression are the same.

[[autodoc]] trainers.text_classification.params.TextClassificationParams

---

Source/Tasks/Token Classification Copy Section

Token Classification

Token classification is the task of classifying each token in a sequence. This can be used
for Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and more. Get your data ready in
proper format and then with just a few clicks, your state-of-the-art model will be ready to
be used in production.

Data Format Copy Chapter

The data should be in the following CSV format:

tokens,tags
"['I', 'love', 'Paris']","['O', 'O', 'B-LOC']"
"['I', 'live', 'in', 'New', 'York']","['O', 'O', 'O', 'B-LOC', 'I-LOC']"
.
.
.

or you can also use JSONL format:

{"tokens": ["I", "love", "Paris"],"tags": ["O", "O", "B-LOC"]}
{"tokens": ["I", "live", "in", "New", "York"],"tags": ["O", "O", "O", "B-LOC", "I-LOC"]}
.
.
.

As you can see, we have two columns in the CSV file. One column is the tokens and the other
is the tags. Both the columns are stringified lists! The tokens column contains the tokens
of the sentence and the tags column contains the tags for each token.

If your CSV is huge, you can divide it into multiple CSV files and upload them separately.
Please make sure that the column names are the same in all CSV files.

One way to divide the CSV file using pandas is as follows:

import pandas as pd
Set the chunk size

chunk_size = 1000
i = 1
Open the CSV file and read it in chunks

for chunk in pd.read_csv('example.csv', chunksize=chunk_size):
    # Save each chunk to a new file
    chunk.to_csv(f'chunk_{i}.csv', index=False)
    i += 1

Sample dataset from HuggingFace Hub: conll2003

Columns Copy Chapter

Your CSV/JSONL dataset must have two columns: tokens and tags.

Parameters Copy Chapter

[[autodoc]] trainers.token_classification.params.TokenClassificationParams

---

Source/Autotrain Api Copy Section

AutoTrain API

With AutoTrain API, you can run your own instance of AutoTrain and use it to
train models on Hugging Face Spaces infrastructure (local training coming soon).
This API is designed to be used with autotrain compatible models and datasets, and it provides a simple interface to
train models with minimal configuration.

Getting Started Copy Chapter

To get started with AutoTrain API, all you need to do is install autotrain-advanced
as discussed in running locally section and run the autotrain app command:

$ autotrain app --port 8000 --host 127.0.0.1

You can then access the API reference at http://127.0.0.1:8000/docs.

Example Usage Copy Chapter

curl -X POST "http://127.0.0.1:8000/api/create_project" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer hf_XXXXX" \
     -d '{
           "username": "abhishek",
           "project_name": "my-autotrain-api-model",
           "task": "llm:orpo",
           "base_model": "meta-llama/Meta-Llama-3-8B-Instruct",
           "hub_dataset": "argilla/distilabel-capybara-dpo-7k-binarized",
           "train_split": "train",
           "hardware": "spaces-a10g-large",
           "column_mapping": {
               "text_column": "chosen",
               "rejected_text_column": "rejected",
               "prompt_text_column": "prompt"
           },
           "params": {
               "block_size": 1024,
               "model_max_length": 4096,
               "max_prompt_length": 512,
               "epochs": 1,
               "batch_size": 2,
               "lr": 0.00003,
               "peft": true,
               "quantization": "int4",
               "target_modules": "all-linear",
               "padding": "right",
               "optimizer": "adamw_torch",
               "scheduler": "linear",
               "gradient_accumulation": 4,
               "mixed_precision": "fp16",
               "chat_template": "chatml"
           }
         }'

---

Source/Col Map Copy Section

Understanding Column Mapping

Column mapping is a critical setup process in AutoTrain that informs the system
about the roles of different columns in your dataset. Whether it's a tabular
dataset, text classification data, or another type, the need for precise
column mapping ensures that AutoTrain processes each dataset element correctly.

How Column Mapping Works Copy Chapter

AutoTrain has no way of knowing what the columns in your dataset represent.
AutoTrain requires a clear understanding of each column's function within
your dataset to train models effectively. This is managed through a
straightforward mapping system in the user interface, represented as a dictionary.
Here's a typical example:

{"text": "text", "label": "target"}

In this example, the text column in your dataset corresponds to the text data
AutoTrain uses for processing, and the target column is treated as the
label for training.

But let's not get confused! AutoTrain has a way to understand what each column in your dataset represents.
If your data is already in AutoTrain format, you dont need to change column mappings.
If not, you can easily map the columns in your dataset to the correct AutoTrain format.

In the UI, you will see column mapping as a dictionary:

{"text": "text", "label": "target"}

Here, the column text in your dataset is mapped to the AutoTrain column text,
and the column target in your dataset is mapped to the AutoTrain column label.

Let's say you are training a text classification model and your dataset has the following columns:

full_text, target_sentiment
"this movie is great", positive
"this movie is bad", negative

You can map these columns to the AutoTrain format as follows:

{"text": "full_text", "label": "target_sentiment"}

If your dataset has the columns: text and label, you don't need to change the column mapping.

Let's take a look at column mappings for each task:

LLM Copy Chapter

Note: For all LLM tasks, if the text column(s) is not formatted i.e. if contains samples in chat format (dict or json), then you
should use chat_template parameter. Read more about it in LLM Parameters Section.

SFT / Generic Trainer Copy Section

{"text": "text"}

text: The column in your dataset that contains the text data.

Reward Trainer Copy Section

{"text": "text", "rejected_text": "rejected_text"}

text: The column in your dataset that contains the text data.

rejected_text: The column in your dataset that contains the rejected text data.

DPO / ORPO Trainer Copy Section

{"prompt": "prompt", "text": "text", "rejected_text": "rejected_text"}

prompt: The column in your dataset that contains the prompt data.

text: The column in your dataset that contains the text data.

rejected_text: The column in your dataset that contains the rejected text data.

Text Classification & Regression, Seq2Seq Copy Chapter

For text classification and regression, the column mapping should be as follows:

{"text": "dataset_text_column", "label": "dataset_target_column"}

text: The column in your dataset that contains the text data.

label: The column in your dataset that contains the target variable.

Token Classification Copy Chapter

{"text": "tokens", "label": "tags"}

text: The column in your dataset that contains the tokens. These tokens must be a list of strings.

label: The column in your dataset that contains the tags. These tags must be a list of strings.

For token classification, if you are using a CSV, make sure that the columns are stringified lists.

Tabular Classification & Regression Copy Chapter

{"id": "id", "label": ["target"]}

id: The column in your dataset that contains the unique identifier for each row.

label: The column in your dataset that contains the target variable. This should be a list of strings.

For a single target column, you can pass a list with a single element.

For multiple target columns, e.g. a multi label classification task, you can pass a list with multiple elements.

Image Classification

For image classification, the column mapping should be as follows:

{"image": "image_column", "label": "label_column"}

Image classification requires column mapping only when you are using a dataset from Hugging Face Hub.
For uploaded datasets, leave column mapping as it is.

Sentence Transformers

For all sentence transformers tasks, one needs to map columns to sentence1_column, sentence2_column, sentence3_column & target_column column.
Not all columns need to be mapped for all trainers of sentence transformers.

pair: pair:')" title="Copy chapter prompt for LLMs"> Copy Chapter

{"sentence1_column": "anchor", "sentence2_column": "positive"}

pair_class: pair_class:')" title="Copy chapter prompt for LLMs"> Copy Chapter

{"sentence1_column": "premise", "sentence2_column": "hypothesis", "target_column": "label"}

pair_score: pair_score:')" title="Copy chapter prompt for LLMs"> Copy Chapter

{"sentence1_column": "sentence1", "sentence2_column": "sentence2", "target_column": "score"}

triplet: triplet:')" title="Copy chapter prompt for LLMs"> Copy Chapter

{"sentence1_column": "anchor", "sentence2_column": "positive", "sentence3_column": "negative"}

qa: qa:')" title="Copy chapter prompt for LLMs"> Copy Chapter

{"sentence1_column": "query", "sentence2_column": "answer"}

Extractive Question Answering

For extractive question answering, the column mapping should be as follows:

{"text": "context", "question": "question", "answer": "answers"}

where answer is a dictionary with keys text and answer_start.

Ensuring Accurate Mapping Copy Chapter

To ensure your model trains correctly:

- Verify Column Names: Double-check that the names used in the mapping dictionary accurately reflect those in your dataset.

- Format Appropriately: Especially in token classification, ensure your data format matches expectations (e.g., lists of strings).

- Update Mappings for New Datasets: Each new dataset might require its unique mappings based on its structure and the task at hand.

By following these guidelines and using the provided examples as templates,
you can effectively instruct AutoTrain on how to interpret and handle your
data for various machine learning tasks. This process is fundamental for
achieving optimal results from your model training endeavors.

---

Source/Config Copy Section

AutoTrain Configs

AutoTrain Configs are the way to use and train models using AutoTrain locally.

Once you have installed AutoTrain Advanced, you can use the following command to train models using AutoTrain config files:

$ export HF_USERNAME=your_hugging_face_username
$ export HF_TOKEN=your_hugging_face_write_token
$ autotrain --config path/to/config.yaml

Example configurations for all tasks can be found in the configs directory of
the AutoTrain Advanced GitHub repository.

Here is an example of an AutoTrain config file:

task: llm
base_model: meta-llama/Meta-Llama-3-8B-Instruct
project_name: autotrain-llama3-8b-orpo
log: tensorboard
backend: local
data:
  path: argilla/distilabel-capybara-dpo-7k-binarized
  train_split: train
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected
params:
  trainer: orpo
  block_size: 1024
  model_max_length: 2048
  max_prompt_length: 512
  epochs: 3
  batch_size: 2
  lr: 3e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 4
  mixed_precision: bf16
hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

In this config, we are finetuning the meta-llama/Meta-Llama-3-8B-Instruct model
on the argilla/distilabel-capybara-dpo-7k-binarized dataset using the orpo
trainer for 3 epochs with a batch size of 2 and a learning rate of 3e-5.
More information on the available parameters can be found in the Data Formats and Parameters section.

In case you dont want to push the model to hub, you can set push_to_hub to false in the config file.
If not pushing the model to hub username and token are not required. Note: they may still be needed
if you are trying to access gated models or datasets.

---

Source/Cost Copy Section

How much does it cost?

AutoTrain offers an accessible approach to model training, providing deployable models
with just a few clicks. Understanding the cost involved is essential to planning and
executing your projects efficiently.

Local Usage Copy Chapter

When you choose to use AutoTrain locally on your own hardware, there is no cost.
This option is ideal for those who prefer to manage their own infrastructure and
do not require the scalability that cloud resources offer.

Using AutoTrain on Hugging Face Spaces Copy Chapter

Pay-As-You-Go: Costs for using AutoTrain in Hugging Face Spaces are based on the
computing resources you consume. This flexible pricing structure ensures you only pay
for what you use, making it cost-effective and scalable for projects of any size.

Ownership and Portability: Unlike some other platforms, AutoTrain does not retain
ownership of your models. Once training is complete, you are free to download and
deploy your models wherever you choose, providing flexibility and control over your all your assets.

Pricing Details Copy Section

Resource-Based Billing: Charges are accrued per minute according to the type of hardware
utilized during training. This means you can scale your resource usage based on the
complexity and needs of your projects.

For a detailed breakdown of the costs associated with using Hugging Face Spaces,
please refer to the pricing section on our website.

To access the paid features of AutoTrain, you must have a valid payment method on file.
You can manage your payment options and view your billing information in
the billing section of your Hugging Face account settings.

By offering both free and flexible paid options, AutoTrain ensures that users can choose
the most suitable model training solution for their needs, whether they are experimenting
on a local machine or scaling up operations on Hugging Face Spaces.

---

Source/Faq Copy Section

Frequently Asked Questions

Are my data and models secure? Copy Chapter

Yes, your data and models are secure. AutoTrain uses the Hugging Face Hub to store your data and models.
All your data and models are uploaded to your Hugging Face account as private repositories and are only accessible by you.
Read more about security here.

Do you upload my data to the Hugging Face Hub? Copy Chapter

AutoTrain will not upload your dataset to the Hub if you are using the local backend or training in the same space.
AutoTrain will push your dataset to the Hub if you are using features like: DGX Cloud
or using local CLI to train on Hugging Face's infrastructure.

You can safely remove the dataset from the Hub after training is complete.
If uploaded, the dataset will be stored in your Hugging Face account as a private repository and will only be accessible by you
and the training process. It is not used once the training is complete.

My training space paused for no reason mid-training Copy Chapter

AutoTrain Training Spaces will pause itself after training is done (or failed). This is done to save resources and costs.
If your training failed, you can still see the space logs and find out what went wrong. Note: you won't be able to retrive the logs if you restart the space.

Another reason for the space to pause is if the space is space's sleep time kicking in. If you have a long running training job, you must set the sleep time to a much higher value.
The space will anyways pause itself after the training is done thus saving you costs.

I get error Your installed package nvidia-ml-py is corrupted. Skip patch functions Your installed package nvidia-ml-py is corrupted. Skip patch functions')" title="Copy chapter prompt for LLMs"> Copy Chapter

This error can be safely ignored. It is a warning from the nvitop library and does not affect the functionality of AutoTrain.

I get 409 conflict error when using the UI Copy Chapter

This error occurs when you try to create a project with the same name as an existing project.
To resolve this error, you can either delete the existing project or create a new project
with a different name.

This error can also occur when you are trying to train a model while a model is already training in the same space or locally.

The model I want to use doesn't show up in the model selection dropdown. Copy Chapter

If the model you want to use is not available in the model selection dropdown,
you can add it in the environment variable AUTOTRAIN_CUSTOM_MODELS in the space settings.
For example, if you want to add the xxx/yyy model, go to space settings, create a variable named AUTOTRAIN_CUSTOM_MODELS
and set the value to xxx/yyy.

You can also pass the model name as query parameter in the URL. For example, if you want to use the xxx/yyy model,
you can use the URL https://huggingface.co/spaces/your_autotrain_space?custom_models=xxx/yyy.

How do I use AutoTrain locally? Copy Chapter

AutoTrain can be used locally by installing the AutoTrain Advanced pypi package.
You can read more in Use AutoTrain Locally section.

Can I run AutoTrain on Colab? Copy Chapter

To start the UI on Colab, you can simply click on the following link:

[](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain.ipynb)

Please note, to run the app on Colab, you will need an ngrok token. You can get one by signing up for free on ngrok.
This is because Colab does not allow exposing ports to the internet directly.

To use the CLI instead on Colab, you can follow the same instructions as for using AutoTrain locally.

Does AutoTrain have a docker image? Copy Chapter

Yes, AutoTrain has a docker image.
You can find the docker image on Docker Hub here.

Is windows supported? Copy Chapter

Unfortunately, AutoTrain does not officially support Windows at the moment.
You can try using WSL (Windows Subsystem for Linux) to run AutoTrain on Windows or the docker image.

"--project-name" argument can not be set as a directory Copy Chapter

--project-name argument should not be a path. it will be created where autotrain command is run.
This parameter must be alphanumeric and can contain hypens.

I am getting config.json not found error config.json not found error')" title="Copy chapter prompt for LLMs"> Copy Chapter

This means you have trained an adapter model (peft=true) which doesnt generate config.json.
It doesnt matter though, the model can still be loaded with AutoModelForCausalLM or with Inference endpoints.
If you want to merge weights with base models, you can use autotrain tools. Please read about it in miscelleneous section.

Does autotrain support multi-gpu training? Copy Chapter

Yes, autotrain supports multi-gpu training.
AutoTrain will determine on its own if the user is running the command on a multi-gpu setup and will use
multi-gpu ddp if number of gpus is greater than 1 and less than 4 and deepspeed if number of gpus is greater than or equal to 4.

How can i use a hub dataset with multiple configs? Copy Chapter

If your hub dataset has multiple configs, you can use train_split parameter to specify the both the config and the split.
For example, in this dataset here,
there are multiple configs: pair, pair-class, pair-score and triplet.

If i want to use train split of pair-class config, i can use write pair-class:train as train_split in the UI or the CLI / config.

An example config is shown below:

data:
  path: sentence-transformers/all-nli
  train_split: pair-class:train
  valid_split: pair-class:test
  column_mapping:
    sentence1_column: premise
    sentence2_column: hypothesis
    target_column: label

---

Source/Index Copy Section

AutoTrain

🤗 AutoTrain Advanced (or simply AutoTrain), developed by Hugging Face, is a robust no-code
platform designed to simplify the process of training state-of-the-art models across
multiple domains: Natural Language Processing (NLP), Computer Vision (CV),
and even Tabular Data analysis. This tool leverages the powerful frameworks created by
various teams at Hugging Face, making advanced machine learning and artificial intelligence accessible to a broader
audience without requiring deep technical expertise.

Who should use AutoTrain? Copy Chapter

AutoTrain is the perfect tool for anyone eager to dive into the world of machine learning
without getting bogged down by the complexities of model training.
Whether you're a business professional, researcher, educator, or hobbyist,
AutoTrain offers the simplicity of a no-code interface while still providing the
capabilities necessary to develop sophisticated models tailored to your unique datasets.

AutoTrain is for anyone who wants to train a state-of-the-art model for a NLP, CV, Speech or even Tabular task,
but doesn't want to spend time on the technical details of training a model.

Our mission is to democratize machine learning technology, ensuring it is not only
accessible to data scientists and ML engineers but also to those without a technical
background. If you're looking to harness the power of AI for your projects,
AutoTrain is your answer.

How to use AutoTrain? Copy Chapter

We offer several ways to use AutoTrain:

- No code users can use AutoTrain Advanced by creating a new space with AutoTrain Docker image:
Click here to create AutoTrain Space.
Remember to keep your space private and ensure it is equipped with the necessary hardware resources (GPU) for optimal performance.

- If you prefer a more hands-on approach, AutoTrain Advanced can also be run locally
through its intuitive UI or accessed via the Python API provided in the autotrain-advanced
package. This flexibility allows developers to integrate AutoTrain capabilities directly
into their projects, customize workflows, and enhance their toolsets with advanced machine
learning functionalities.

By bridging the gap between cutting-edge technology and practical usability,
AutoTrain Advanced empowers users to achieve remarkable results in AI without the need
for extensive programming knowledge. Start your journey with AutoTrain today and unlock
the potential of machine learning for your projects!

Walkthroughs Copy Chapter

To get started with AutoTrain, check out our walkthroughs and tutorials:

- Extractive Question Answering with AutoTrain
- Finetuning PaliGemma with AutoTrain
- Training an Object Detection Model with AutoTrain
- How to Fine-Tune Custom Embedding Models Using AutoTrain
- Train Custom Models on Hugging Face Spaces with AutoTrain SpaceRunner
- How to Finetune phi-3 on MacBook Pro
- Finetune Mixtral 8x7B with AutoTrain
- Easily Train Models with H100 GPUs on NVIDIA DGX Cloud

---

Source/Quickstart Copy Section

Quickstart Guide for Local Training

This quickstart is for local installation and usage.
If you want to use AutoTrain on Hugging Face Spaces, please refer to the AutoTrain on Hugging Face Spaces section.

You can install AutoTrain Advanced using pip:

$ pip install autotrain-advanced

It is advised to install autotrain-advanced in a virtual environment to avoid any conflicts with other packages.
Note: AutoTrain doesn't install pytorch, torchaudio, torchvision, or any other large dependencies. You will need to install them separately.

$ conda create -n autotrain python=3.10
$ conda activate autotrain
$ pip install autotrain-advanced
$ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
$ conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc
$ conda install xformers -c xformers
$ python -m nltk.downloader punkt
$ pip install flash-attn --no-build-isolation # if you want to use flash-attn
$ pip install deepspeed # if you want to use deepspeed

$ export HF_TOKEN=your_hugging_face_write_token
$ autotrain app --host 127.0.0.1 --port 8000

This will start the app on http://127.0.0.1:8000.

Using AutoTrain Command Line Interface (CLI)

It is also possible to use the CLI:

$ export HF_TOKEN=your_hugging_face_write_token
$ autotrain --help

This will show the CLI commands that can be used:

usage: autotrain <command> [<args>]
positional arguments:
{
    app,
    llm,
    setup,
    api,
    text-classification,
    text-regression,
    image-classification,
    tabular,
    spacerunner,
    seq2seq,
    token-classification
}
    
    commands
options:
  -h, --help            show this help message and exit
  --version, -v         Display AutoTrain version
  --config CONFIG       Optional configuration file
For more information about a command, run: autotrain <command> --help

It is advised to use only the autotrain --config CONFIG_FILE command for training when using the CLI.

The autotrain commands that end users will be interested in are:

- app: Start the AutoTrain UI
- llm: Train a language model
- text-classification: Train a text classification model
- text-regression: Train a text regression model
- image-classification: Train an image classification model
- tabular: Train a tabular model
- spacerunner: Train any custom model using SpaceRunner
- seq2seq: Train a sequence-to-sequence model
- token-classification: Train a token classification model

Note: above commands are not required if you use preferred autotrain --config CONFIG_FILE command to train the models.

---

Source/Quickstart Py Copy Section

Quickstart with Python

AutoTrain is a library that allows you to train state of the art models on Hugging Face Spaces, or locally.
It provides a simple and easy-to-use interface to train models for various tasks like llm finetuning, text classification,
image classification, object detection, and more.

In this quickstart guide, we will show you how to train a model using AutoTrain in Python.

Getting Started Copy Chapter

AutoTrain can be installed using pip:

$ pip install autotrain-advanced

The example code below shows how to finetune an LLM model using AutoTrain in Python:

import os
from autotrain.params import LLMTrainingParams
from autotrain.project import AutoTrainProject

params = LLMTrainingParams(
    model="meta-llama/Llama-3.2-1B-Instruct",
    data_path="HuggingFaceH4/no_robots",
    chat_template="tokenizer",
    text_column="messages",
    train_split="train",
    trainer="sft",
    epochs=3,
    batch_size=1,
    lr=1e-5,
    peft=True,
    quantization="int4",
    target_modules="all-linear",
    padding="right",
    optimizer="paged_adamw_8bit",
    scheduler="cosine",
    gradient_accumulation=8,
    mixed_precision="bf16",
    merge_adapter=True,
    project_name="autotrain-llama32-1b-finetune",
    log="tensorboard",
    push_to_hub=True,
    username=os.environ.get("HF_USERNAME"),
    token=os.environ.get("HF_TOKEN"),
)

backend = "local"
project = AutoTrainProject(params=params, backend=backend, process=True)
project.create()

In this example, we are finetuning the meta-llama/Llama-3.2-1B-Instruct model on the HuggingFaceH4/no_robots dataset.
We are training the model for 3 epochs with a batch size of 1 and a learning rate of 1e-5.
We are using the paged_adamw_8bit optimizer and the cosine scheduler.
We are also using mixed precision training with a gradient accumulation of 8.
The final model will be pushed to the Hugging Face Hub after training.

To train the model, run the following command:

$ export HF_USERNAME=<your-hf-username>
$ export HF_TOKEN=<your-hf-write-token>
$ python train.py

This will create a new project directory with the name autotrain-llama32-1b-finetune and start the training process.
Once the training is complete, the model will be pushed to the Hugging Face Hub.

Your HF_TOKEN and HF_USERNAME are only required if you want to push the model or if you are accessing a gated model or dataset.

AutoTrainProject Class Copy Chapter

[[autodoc]] project.AutoTrainProject

Parameters Copy Chapter

Text Tasks Copy Section

[[autodoc]] trainers.clm.params.LLMTrainingParams

[[autodoc]] trainers.sent_transformers.params.SentenceTransformersParams

[[autodoc]] trainers.seq2seq.params.Seq2SeqParams

[[autodoc]] trainers.token_classification.params.TokenClassificationParams

[[autodoc]] trainers.extractive_question_answering.params.ExtractiveQuestionAnsweringParams

[[autodoc]] trainers.text_classification.params.TextClassificationParams

[[autodoc]] trainers.text_regression.params.TextRegressionParams

Image Tasks Copy Section

[[autodoc]] trainers.image_classification.params.ImageClassificationParams

[[autodoc]] trainers.image_regression.params.ImageRegressionParams

[[autodoc]] trainers.object_detection.params.ObjectDetectionParams

Tabular Tasks Copy Section

[[autodoc]] trainers.tabular.params.TabularParams

---

Source/Quickstart Spaces Copy Section

Quickstart Guide to AutoTrain on Hugging Face Spaces

AutoTrain on Hugging Face Spaces is the preferred choice for a streamlined experience in
model training. This platform is optimized for ease of use, with pre-installed dependencies
and managed hardware resources. AutoTrain on Hugging Face Spaces can be used both by
no-code users and developers, making it versatile for various levels of expertise.

Creating a New AutoTrain Space Copy Chapter

Getting started with AutoTrain is straightforward. Here’s how you can create your new space:

1. Visit the AutoTrain Page: To create a new space with AutoTrain Docker image, all you need to do is go
to AutoTrain Homepage and click on "Create new project".

2. Log In or View the Setup Screen: If not logged in, you'll be prompted to do so. Then, you’ll see a screen similar to this:

3. Set Up Your Space:

- Choose a Space Name: Name your space something relevant to your project.

- Allocate Hardware Resources: Select the necessary computational resources based on your project needs.

- Duplicate Space: Click on "Duplicate Space" to initiate your AutoTrain space with the Docker image.

4. Configuration Options:

- PAUSE_ON_FAILURE: Set this to 0 if you prefer the space not to pause on training failures, useful for running continuous experiments. This option can also be used if you continuously want to perfom many experiments in the same space.

5. Launch and Train:

- Once done, in a few seconds, the AutoTrain Space will be up and running and you will be presented with the following screen:

- From here, you can select tasks, upload datasets, choose models, adjust hyperparameters (if needed),
and start the training process directly within the space.

- The space will manage its own activity, shutting down post-training unless configured
otherwise based on the PAUSE_ON_FAILURE setting.

6. Monitoring Progress:

- All training logs and progress can be monitored via TensorBoard, accessible under
username/project_name on the Hugging Face Hub.

- Once training concludes successfully, you’ll find the model files in the same repository.

7. Navigating the UI:

- If you need help understanding any UI elements, click on the small (i) information icons for detailed descriptions.

If you are confused about the UI elements, click on the small (i) information icon to get more information about the UI element.

For data formats and detailed parameter information, please see the Data Formats and Parameters section where we provide
example datasets and detailed information about the parameters for each task supported by AutoTrain.

Ensuring Your AutoTrain is Up-to-Date Copy Chapter

We are constantly adding new features and tasks to AutoTrain Advanced. To benefit from the latest features, tasks, and bug fixes, update your AutoTrain space regularly:

- Factory Reboot: Navigate to the settings page of your space and click on "Factory reboot" to upgrade to the latest version of AutoTrain Advanced.

- Note: Simply "restarting" the space does not update it; a factory reboot is necessary for a complete update.


For additional details on data formats and specific parameters, refer to the
'Data Formats and Parameters' section where we provide example datasets and extensive
parameter information for each supported task by AutoTrain.


With these steps, you can effortlessly initiate and manage your AutoTrain projects on
Hugging Face Spaces, leveraging the platform's robust capabilities for your machine learning and AI
needs.

---

Source/Support Copy Section

Help and Support

If you need assistance with AutoTrain Advanced or have questions about your projects,
you can reach out through several dedicated support channels. We're here to help you
navigate any issues you encounter, from technical queries to billing concerns.
Below are the best ways to get support:


- For technical support or to report a bug, you can create an issue
directly in the AutoTrain Advanced GitHub repository. GitHub repo is ideal for tracking bugs,
requesting features, or getting help with troubleshooting problems. When submitting an
issue, please include all the details in question to help us provide the most
relevant support quickly.

- Ask in the Hugging Face Forum. This space is perfect for asking questions,
sharing your experiences, or discussing AutoTrain with other users and the Hugging Face
team. The forum is a great resource for getting advice, learning best practices, and
connecting with other machine learning practitioners.

- For enterprise users or specific inquiries related to billing, please email us directly.
This channel ensures that your more sensitive or account-specific issues are handled
appropriately and confidentially. When emailing, please provide your username and
project name so we can assist you efficiently.

Please note: e-mail support is only available for pro/enterprise users or those with specific queries about billing.


By utilizing these support channels, you can ensure that any hurdles you face while using
AutoTrain Advanced are addressed promptly, allowing you to focus on achieving your project
goals. Whether you're a beginner or an experienced user, we are here to support your
journey in AI model training.

---

README Copy Section

🤗 AutoTrain Advanced

AutoTrain Advanced: faster and easier training and deployments of state-of-the-art machine learning models. AutoTrain Advanced is a no-code solution that allows you to train machine learning models in just a few clicks. Please note that you must upload data in correct format for project to be created. For help regarding proper data format and pricing, check out the documentation.

NOTE: AutoTrain is free! You only pay for the resources you use in case you decide to run AutoTrain on Hugging Face Spaces. When running locally, you only pay for the resources you use on your own infrastructure.

Supported Tasks Copy Chapter

| Task | Status | Python Notebook | Example Configs |
| --- | --- | --- | --- |
| LLM SFT Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | llm_sft_finetune.yaml |
| LLM ORPO Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | llm_orpo_finetune.yaml |
| LLM DPO Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | llm_dpo_finetune.yaml |
| LLM Reward Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | llm_reward_finetune.yaml |
| LLM Generic/Default Finetuning | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb) | llm_generic_finetune.yaml |
| Text Classification | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/text_classification.ipynb) | text_classification.yaml |
| Text Regression | ✅ | [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/notebooks/text_regression.ipynb) | text_regression.yaml |
| Token Classification | ✅ | Coming Soon | token_classification.yaml |
| Seq2Seq | ✅ | Coming Soon | seq2seq.yaml |
| Extractive Question Answering | ✅ | Coming Soon | extractive_qa.yaml |
| Image Classification | ✅ | Coming Soon | image_classification.yaml |
| Image Scoring/Regression | ✅ | Coming Soon | image_regression.yaml |
| VLM | 🟥 | Coming Soon | vlm.yaml |

Running UI on Colab or Hugging Face Spaces Copy Chapter

- Deploy AutoTrain on Hugging Face Spaces: [](https://huggingface.co/login?next=%2Fspaces%2Fautotrain-projects%2Fautotrain-advanced%3Fduplicate%3Dtrue)


- Run AutoTrain UI on Colab via ngrok: [](https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_ngrok.ipynb)

Local Installation Copy Chapter

You can Install AutoTrain-Advanced python package via PIP. Please note you will need python >= 3.10 for AutoTrain Advanced to work properly.

pip install autotrain-advanced

Please make sure that you have git lfs installed. Check out the instructions here: https://github.com/git-lfs/git-lfs/wiki/Installation

You also need to install torch, torchaudio and torchvision.

The best way to run autotrain is in a conda environment. You can create a new conda environment with the following command:

conda create -n autotrain python=3.10
conda activate autotrain
pip install autotrain-advanced
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc

Once done, you can start the application using:

autotrain app --port 8080 --host 127.0.0.1


If you are not fond of UI, you can use AutoTrain Configs to train using command line or simply AutoTrain CLI.

To use config file for training, you can use the following command:

autotrain --config <path_to_config_file>


You can find sample config files in the configs` directory of this repository.

Example config file for finetuning SmolLM2:

task: llm-sft
base_model: HuggingFaceTB/SmolLM2-1.7B-Instruct
project_name: autotrain-smollm2-finetune
log: tensorboard
backend: local
data:
  path: HuggingFaceH4/no_robots
  train_split: train
  valid_split: null
  chat_template: tokenizer
  column_mapping:
    text_column: messages
params:
  block_size: 2048
  model_max_length: 4096
  epochs: 2
  batch_size: 1
  lr: 1e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: paged_adamw_8bit
  scheduler: linear
  gradient_accumulation: 8
  mixed_precision: bf16
  merge_adapter: true
hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

To fine-tune a model using the config file above, you can use the following command:

$ export HF_USERNAME=<your_hugging_face_username>
$ export HF_TOKEN=<your_hugging_face_write_token>
$ autotrain --config <path_to_config_file>

Documentation Copy Chapter

Documentation is available at https://hf.co/docs/autotrain/

Citation Copy Chapter

/ Detailed source-code truncated for AI context efficiency. /

---