h2o-llmstudio Documentation

File: README.md Copy Chapter

Welcome to H2O LLM Studio, a framework and no-code GUI designed for fine-tuning state-of-the-art large language models (LLMs).

Jump to Copy Chapter

• With H2O LLM Studio, you can
• Quickstart
• What's New
• Setup
  • Recommended Install
  • Virtual Environments
• Run H2O LLM Studio GUI
• Run H2O LLM Studio GUI using Docker
• Run H2O LLM Studio with command line interface (CLI)
• Troubleshooting
• Data format and example data
• Training your model
• Example: Run on OASST data via CLI
• Model checkpoints
• Documentation
• Contributing
• License

With H2O LLM Studio, you can Copy Chapter

• easily and effectively fine-tune LLMs without the need for any coding experience.
• use a graphical user interface (GUI) specially designed for large language models.
• fine-tune any LLM using a large variety of hyperparameters.
• use recent fine-tuning techniques such as Low-Rank Adaptation (LoRA) and 8-bit model training with a low memory footprint.
• use Reinforcement Learning (RL) to fine-tune your model (experimental).
• use advanced evaluation metrics to judge generated answers by the model.
• track and compare your model performance visually. In addition, W&B integration can be used.
• chat with your model and get instant feedback on your model performance.
• easily export your model to the Hugging Face Hub and share it with the community.

Quickstart Copy Chapter

For questions, discussing, or just hanging out, come and join our Discord!

Use cloud-based runpod.io instance to run the latest version of H2O LLM Studio with GUI.

Using CLI for fine-tuning LLMs:

What's New Copy Chapter

• PR 788 New problem type for Causal Regression Modeling allows to train single target regression data using LLMs.
• PR 747 Fully removed RLHF in favor of DPO/IPO/KTO optimization.
• PR 741 Removing separate max length settings for prompt and answer in favor of a single max_length settings better resembling chat_template functionality from transformers.
• PR 592 Added KTOPairLoss for DPO modeling allowing to train models with simple preference data. Data currently needs to be manually prepared by randomly matching positive and negative examples as pairs.
• PR 592 Starting to deprecate RLHF in favor of DPO/IPO optimization. Training is disabled, but old experiments are still viewable. RLHF will be fully removed in a future release.
• PR 530 Introduced a new problem type for DPO/IPO optimization. This optimization technique can be used as an alternative to RLHF.
• PR 288 Introduced DeepSpeed for sharded training allowing to train larger models on machines with multiple GPUs. Requires NVLink. This feature replaces FSDP and offers more flexibility. DeepSpeed requires a system installation of CUDA Toolkit and we recommend using version 12.1. See Recommended Install.
• PR 449 New problem type for Causal Classification Modeling allows to train binary and multiclass models using LLMs.
• PR 364 User secrets are now handled more securely and flexible. Support for handling secrets using the 'keyring' library was added. User settings are tried to be migrated automatically.

Please note that due to current rapid development we cannot guarantee full backwards compatibility of new functionality. We thus recommend to pin the version of the framework to the one you used for your experiments. For resetting, please delete/backup your data and output folders.

Setup Copy Chapter

H2O LLM Studio requires a machine with Ubuntu 16.04+ and at least one recent NVIDIA GPU with NVIDIA drivers version >= 470.57.02. For larger models, we recommend at least 24GB of GPU memory.

For more information about installation prerequisites, see the Set up H2O LLM Studio guide in the documentation.

For a performance comparison of different GPUs, see the H2O LLM Studio performance guide in the documentation.

Recommended Install Copy Section

The recommended way to install H2O LLM Studio is using uv with Python 3.10. To install Python 3.10 on Ubuntu 20.04+, execute the following commands:

Installing NVIDIA Drivers (if required)

If deploying on a 'bare metal' machine running Ubuntu, one may need to install the required NVIDIA drivers and CUDA. The following commands show how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an example. One can update the following based on their OS.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-4

Virtual environments Copy Section

We offer various ways of setting up the necessary python environment.

UV virtual environment

The following command will create a virtual environment using uv and will install the dependencies:

make setup

Run H2O LLM Studio GUI Copy Chapter

You can start H2O LLM Studio using the following command:

make llmstudio

This command will start the H2O Wave server and app.
Navigate to http://localhost:10101/ (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models!

If you are running H2O LLM Studio with a custom environment other than uv, you need to start the app as follows:

H2O_WAVE_MAX_REQUEST_SIZE=25MB \
H2O_WAVE_NO_LOG=true \
H2O_WAVE_PRIVATE_DIR="/download/@output/download" \
wave run llm_studio.app

Run H2O LLM Studio GUI using Docker Copy Chapter

Install Docker first by following instructions from NVIDIA Containers. Make sure to have nvidia-container-toolkit installed on your machine as outlined in the instructions.

H2O LLM Studio images are stored in the h2oai Docker Hub container repository.

mkdir -p `pwd`/llmstudio_mnt
chmod 777 `pwd`/llmstudio_mnt

# make sure to pull latest image if you still have a prior version cached
docker pull h2oairelease/h2oai-llmstudio-app:latest

# run the container
docker run \
    --runtime=nvidia \
    --shm-size=64g \
    --init \
    --rm \
    -it \
    -u `id -u`:`id -g` \
    -p 10101:10101 \
    -v `pwd`/llmstudio_mnt:/mount \
    h2oairelease/h2oai-llmstudio-app:latest

Navigate to http://localhost:10101/ (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models!

(Note other helpful docker commands are docker ps and docker kill.)

Run H2O LLM Studio with command line interface (CLI) Copy Chapter

You can also use H2O LLM Studio with the command line interface (CLI) and specify the configuration .yaml file that contains all the experiment parameters. To fine-tune using H2O LLM Studio with CLI use the following command:

uv run python llm_studio/train.py -Y {path_to_config_yaml_file}

To run on multiple GPUs in DDP mode, run the following command:

bash distributed_train.sh {NR_OF_GPUS} -Y {path_to_config_yaml_file}

By default, the framework will run on the first k GPUs. If you want to specify specific GPUs to run on, use the CUDA_VISIBLE_DEVICES environment variable before the command.

To start an interactive chat with your trained model, use the following command:

uv run python llm_studio/prompt.py -e {experiment_name}

where experiment_name is the output folder of the experiment you want to chat with (see configuration).
The interactive chat will also work with model that were fine-tuned using the UI.

To publish the model to Hugging Face, use the following command:

uv run python llm_studio/publish_to_hugging_face.py -p {path_to_experiment} -d {device} -a {api_key} -u {user_id} -m {model_name} -s {safe_serialization}

path_to_experiment is the output folder of the experiment.
device is the target device for running the model, either 'cpu' or 'cuda:0'. Default is 'cuda:0'.
api_key is the Hugging Face API Key. If the user is logged in, it can be omitted.
user_id is the Hugging Face user ID. If the user is logged in, it can be omitted.
model_name is the name of the model to be published on Hugging Face. It can be omitted.
safe_serialization is a flag indicating whether safe serialization should be used. Default is True.

Troubleshooting Copy Chapter

If running on cloud-based machines such as runpod, you may need to set the following environment variable to allow the H2O Wave server to accept connections from the proxy:

H2O_WAVE_ALLOWED_ORIGINS="*"

If you are experiencing timeouts when running the H2O Wave server remotely, you can increase the timeout by setting the following environment variables:

H2O_WAVE_APP_CONNECT_TIMEOUT="15"
H2O_WAVE_APP_WRITE_TIMEOUT="15"
H2O_WAVE_APP_READ_TIMEOUT="15"
H2O_WAVE_APP_POOL_TIMEOUT="15"

All default to 5 (seconds). Increase them if you are experiencing timeouts. Use -1 to disable the timeout.

Data format and example data Copy Chapter

For details on the data format required when importing your data or example data that you can use to try out H2O LLM Studio, see Data format in the H2O LLM Studio documentation.

Training your model Copy Chapter

With H2O LLM Studio, training your large language model is easy and intuitive. First, upload your dataset and then start training your model. Start by creating an experiment. You can then monitor and manage your experiment, compare experiments, or push the model to Hugging Face to share it with the community.

Example: Run on OASST data via CLI Copy Chapter

As an example, you can run an experiment on the OASST data via CLI. For instructions, see Run an experiment on the OASST data guide in the H2O LLM Studio documentation.

Model checkpoints Copy Chapter

All open-source datasets and models are posted on H2O.ai's Hugging Face page and our H2OGPT repository.

Documentation Copy Chapter

Detailed documentation and frequently asked questions (FAQs) for H2O LLM Studio can be found at https://docs.h2o.ai/h2o-llmstudio/. If you wish to contribute to the docs, navigate to the /documentation folder of this repo and refer to the README.md for more information.

Contributing Copy Chapter

We are happy to accept contributions to the H2O LLM Studio project. Please refer to the CONTRIBUTING.md file for more information.

License Copy Chapter

H2O LLM Studio is licensed under the Apache 2.0 license. Please see the LICENSE file for more information.

File: documentation/docs/get-started/core-features.md Copy Chapter

Core features

No-code fine-tuning Copy Chapter

NLP practitioners can easily fine-tune models without the need for code expertise. The user interface, which is specifically designed for LLMs, allows users to upload large datasets easily and configure hyperparameters to fine-tune the model.

Highly customizable (wide range of hyperparameters) Copy Chapter

H2O LLM Studio supports a wide variety of hyperparameters that can be used to fine-tune the model and supports the following fine-tuning techniques to enable advanced customization:

• Low-Rank Adaptation (LoRA)
• 8-bit model training with a low memory footprint

Advanced evaluation metrics and experiment comparison Copy Chapter

Advanced evaluation metrics in H2O LLM Studio can be used to validate the answers generated by the LLM. This helps to make data-driven decisions about the model. It also offers visual tracking and comparison of experiment performance, making it easy to analyze and compare different fine-tuned models.You can also visualize how different parameters affect the model performance, and optionally use the W&B integration to track and log your experiments.

Instant publishing models Copy Chapter

H2O LLM Studio enables easy model sharing with the community by allowing you to export the model to the Hugging Face Hub with a single click.

Instant feedback on model performance Copy Chapter

Additionally, H2O LLM Studio lets you chat with the fine-tuned model and receive instant feedback about model performance.

File: documentation/docs/get-started/llm-studio-flow.md Copy Chapter

Model flow

The flow of creating and fine-tuning large language models using H2O LLM Studio can be summarized in the following sequential steps:

• Step 1: Import a dataset
• Step 2: Create an experiment
• Step 3: Monitor an experiment
• Step 4: Compare experiments
• Step 5: Export a model to Hugging Face Hub

Step 1: Import a dataset Copy Chapter

As the first step in the experiment flow, prep your data and import your dataset to H2O LLM Studio.

• To learn about supported data connectors and data format, see Supported data connectors and format.
• To learn about how to import a dataset to H2O LLM Studio, see Import a dataset.
• To learn about reviewing and editing a dataset, see View and manage dataset.

Step 2: Create an experiment Copy Chapter

As the second step in the experiment flow, create an experiment using the imported dataset. H2O LLM Studio offers several hyperparameter settings that you can adjust for your experiment model. To ensure that your training process is effective, you may need to specify the hyperparameters like learning rate, batch size, and the number of epochs. H2O LLM Studio provides an overview of all the parameters you’ll need to specify for your experiment.

• To learn about creating a new experiment, see Create an experiment.
• To learn about the settings available for creating an experiment, see Experiment settings.

Step 3: Monitor an experiment Copy Chapter

As the third step in the experiment flow, monitor the launched experiment. H2O LLM Studio allows you to inspect your experiment (model) during and after model training. Simple interactive graphs in H2O LLM Studio allow you to understand the impact of selected hyperparameter values during and after model training. You can then adjust the hyperparameters to further optimize model performance.

To learn about viewing and monitoring an experiment, see View and manage experiments.

Step 4: Compare experiments Copy Chapter

The H2O LLM studio provides a useful feature that allows comparing various experiments and analyzing how different model parameters affect model performance. This feature is a powerful tool for fine-tuning your machine-learning models and ensuring they meet your desired performance metrics.

To learn about comparing multiple experiments, see Compare experiments.

Step 5: Export a model to Hugging Face Hub Copy Chapter

As the final step in the experiment flow, you can export the fine-tuned model to Hugging Face with a single click.

To learn about exporting a trained model to Hugging Face Hub, see, Export trained model to Hugging Face.

File: documentation/docs/get-started/llm-studio-performance.md Copy Chapter

H2O LLM Studio performance

Setting up and running H2O LLM Studio requires the following minimal prerequisites. This page lists out the speed and performance metrics of H2O LLM Studio based on different hardware setups.

The following metrics were measured.

• Hardware setup: The type and number of computing devices used to train the model.
• LLM backbone: The underlying architecture of the language model. For more information, see LLM backbone.
• Quantization: A technique used to reduce the size and memory requirements of the model. For more information, see Quantization.
• Train: The amount of time it took to train the model in hours and minutes.
• Validation: The amount of time it took to validate the mode in hours and minutes.

:::info
The runtimes were gathered using the default parameters.

Expand to see the default parameters

/* Detailed source-code truncated for AI context efficiency. */

:::

File: documentation/docs/get-started/set-up-llm-studio.md Copy Chapter

import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";

Set up H2O LLM Studio

This page guides you through setting up and installing H2O LLM Studio on your local system.

First, download the H2O LLM Studio package from the H2O LLM Studio Github repository. You can use git clone or navigate to the releases page and download the .zip file found within the Assets of the relevant release.

Prerequisites Copy Chapter

H2O LLM Studio requires the following minimum requirements:

• A machine with Ubuntu 16.04+ with atleast one recent Nvidia GPU
• Have at least 128GB+ of system RAM. Larger models and complex tasks may require 256GB+ or more.
• Nvidia drivers v470.57.02 or a later version
• Access to the following URLs:
  • developer.download.nvidia.com
  • pypi.org
  • huggingface.co
  • download.pytorch.org
  • cdn-lfs.huggingface.co

:::info Notes

• Atleast 24GB of GPU memory is recommended for larger models.
• For more information on performance benchmarks based on the hardware setup, see H2O LLM Studio performance.
• The required URLs are accessible by default when you start a GCP instance, however, if you have network rules or custom firewalls in place, it is recommended to confirm that the URLs are accessible before running make setup.
  :::

Installation Copy Chapter

:::note Installation methods

The recommended way to install H2O LLM Studio is using `uv` with Python
  3.10. To install Python 3.10 on Ubuntu 16.04+, execute the following
  commands.


  **Install NVIDIA drivers (if required)**
  
  If you are deploying on a 'bare metal' machine running Ubuntu, you may need
  to install the required Nvidia drivers and CUDA. The following commands show
  how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an
  example. You can update the following based on your respective operating system.


  <code>
    wget
    https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin{" "}
    
    sudo mv cuda-ubuntu2004.pin
    /etc/apt/preferences.d/cuda-repository-pin-600 
    wget
    https://developer.download.nvidia.com/compute/cuda/11.4.3/local_installers/cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb{" "}
    
    sudo dpkg -i
    cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb 
    sudo apt-key add /var/cuda-repo-ubuntu2004-11-4-local/7fa2af80.pub 
    sudo apt-get -y update 
    sudo apt-get -y install cuda
  </code>


  **Create virtual environment (uv) **
  
  The following command creates a virtual environment using `uv` and will install
  the dependencies.
  
    `make setup`

Follow the steps below to install H2O LLM Studio on a Windows machine
  using Windows Subsystem for Linux{" "}
  [WSL2](https://learn.microsoft.com/en-us/windows/wsl/)


  1. Download the{" "}
  <a href="https://www.nvidia.com/download/index.aspx">
    latest nvidia driver
  </a>{" "}
  for Windows.{" "}


  2. Open PowerShell or a Windows Command Prompt window in administrator
  mode.{" "}


  3. Run the following command to confirm that the driver is installed
  properly and see the driver version.
  
    `nvidia-smi`
  


  4. Run the following command to install WSL2.
  
    `wsl --install`
  

5. Launch the WSL2 Ubuntu installation. 

  6. Install the{" "}
  <a href="https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0">
    WSL2 Nvidia Cuda Drivers
  </a>
  .
  
    <code>
      wget
      https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin{" "}
      
      sudo mv cuda-ubuntu2004.pin
      /etc/apt/preferences.d/cuda-repository-pin-600 
      wget
      https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb{" "}
      
      sudo dpkg -i cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb 
      sudo cp /var/cuda-repo-wsl-ubuntu-12-2-local/cuda-*-keyring.gpg
      /usr/share/keyrings/ 
      sudo apt-get update 
      sudo apt-get -y install cuda
    </code>
  


  7. Set up the required python system installs (Python 3.10).
  
    <code>
      sudo add-apt-repository ppa:deadsnakes/ppa 
      sudo apt install python3.10 
      sudo apt-get install python3.10-distutils 
      curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
    </code>
  


  8. Create the virtual environment.
  
    <code>
      sudo apt install -y python3.10-venv
      python3 -m venv llmstudio
      source llmstudio/bin/activate
    </code>
  


  9.Clone the H2O LLM Studio repository locally.
  
    <code>
      git clone https://github.com/h2oai/h2o-llmstudio.git
      cd h2o-llmstudio
    </code>
  


  10. Install H2O LLM Studio dependencies.
  
    `make setup`
  


  11. Run the H2O LLM Studio application.
  
    <code>
      H2O_WAVE_MAX_REQUEST_SIZE=25MB \ 
      H2O_WAVE_NO_LOG=True \ 
      H2O_WAVE_PRIVATE_DIR="/download/@output/download" \ 
      wave run llm_studio.app
    </code>
  


  This will start the H2O Wave server and the H2O LLM Studio app. Navigate
  to <a>http://localhost:10101/</a> (we recommend using Chrome) to access
  H2O LLM Studio and start fine-tuning your models.

Install custom package Copy Chapter

If required, you can install additional Python packages into your environment. You can directly install the custom package by running the following command.

uv add package_name

Run H2O LLM Studio Copy Chapter

There are several ways to run H2O LLM Studio depending on your requirements.

1. Run H2O LLM Studio GUI
2. Run using Docker
3. Run by building your own Docker image
4. Run with the CLI (command-line interface)

Run H2O LLM Studio GUI Copy Section

Run the following command to start the H2O LLM Studio.

make llmstudio

This will start the H2O Wave server and the H2O LLM Studio app. Navigate to http://localhost:10101/ (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models.

If you are running H2O LLM Studio with a custom environment other than uv, start the app as follows:

H2O_WAVE_MAX_REQUEST_SIZE=25MB \
H2O_WAVE_NO_LOG=True \
H2O_WAVE_PRIVATE_DIR="/download/@output/download" \
wave run llm_studio.app

Run using Docker Copy Section

First, install Docker by following the instructions from the NVIDIA Container Installation Guide. H2O LLM Studio images are stored in the h2oai dockerhub container repository.

mkdir -p `pwd`/llmstudio_mnt
docker run \
    --runtime=nvidia \
    --shm-size=64g \
    --init \
    --rm \
    -it \
    -p 10101:10101 \
    -v `pwd`/llmstudio_mnt:/mount \
    h2oairelease/h2oai-llmstudio-app:latest

Navigate to http://localhost:10101/ (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models.

:::info
Other helpful docker commands are docker ps and docker kill.
:::

Run by building your own Docker image Copy Section

docker build -t h2o-llmstudio .
mkdir -p `pwd`/llmstudio_mnt
docker run \
    --runtime=nvidia \
    --shm-size=64g \
    --init \
    --rm \
    -it \
    -p 10101:10101 \
    -v `pwd`/llmstudio_mnt:/mount \
    h2o-llmstudio

Run with command line interface (CLI) Copy Section

You can also use H2O LLM Studio with the command line interface (CLI) and specify the configuration .yaml file that contains all the experiment parameters. To finetune using H2O LLM Studio with CLI and to specify the path to the configuration file that contains the experiment parameters, run:

uv run python llm_studio/train.py -Y {path_to_config_yaml_file}

To run on multiple GPUs in DDP mode, run:

bash distributed_train.sh {NR_OF_GPUS} -Y {path_to_config_yaml_file}

:::info
By default, the framework will run on the first k GPUs. If you want to specify specific GPUs to run on, use the CUDA_VISIBLE_DEVICES environment variable before the command.
:::

To start an interactive chat with your trained model, run:

uv run python llm_studio/prompt.py -e {experiment_name}

experiment_name is the output folder of the experiment you want to chat with. The interactive chat will also work with models that were fine-tuned using the GUI.

File: documentation/docs/get-started/videos.md Copy Chapter

import ReactPlayer from 'react-player'

Videos

Discovering the Potential of LLMs Copy Chapter

:::info Note
In this video, Andreea Turcu delves in-depth into the world of language models, showcasing how users can use H2O.ai's LLM Studio to their full advantage.
:::

The Fine Art of Fine-Tuning Large Language Models Copy Chapter

:::info Note
In this video, Pascal Pfeiffer, Principal Data Scientist at H2O.ai and Kaggle Grandmaster, announces the release of H2O LLM Studio and talks about fine-tuning LLMs using H2O LLM Studio at H2O World India 2023.
:::

Basic introduction to H2O LLM Studio Copy Chapter

:::info Note
In this video, Avkash Chauhan, founder of Prodramp Inc, gives a basic introduction about H2O LLM Studio.
:::

LLM Fine-Tuning, Falcon 40b, and the State of Open-Source Copy Chapter

:::info Note
In this video, Pascal Pfeiffer, the Principal Data Scientist at h2o.ai is interviewed about LLM fine-tuning, being a Kaggle Grandmaster, H2O.ai, Falcon 40b, the state of open-source, and more.
:::

File: documentation/docs/get-started/what-is-h2o-llm-studio.md Copy Chapter

What is H2O LLM Studio?

H2O LLM Studio is an open-source, no-code LLM graphical user interface (GUI) designed for fine-tuning state-of-the-art large language models.

Fine-tuning a pretrained language model requires coding expertise and extensive knowledge about the model and its hyperparameters, however H2O LLM Studio enables NLP practitioners to fine-tune their LLMs easily with no need for coding and better flexibility over customization.

H2O LLM Studio also lets you chat with the fine-tuned model and receive instant feedback about model performance.

Who is H2O LLM Studio for? Copy Chapter

H2O LLM Studio is a free and open-source tool that is designed for anyone who wants to fine-tune their own language models. It is designed to be easy to use and accessible to everyone regardless of their technical expertise.

NLP practitioners and data scientists in particular may find it useful to easily and effectively create and fine-tune large language models.

File: documentation/docs/guide/experiments/compare-experiments.md Copy Chapter

Compare experiments

Using H2O LLM Studio, you can compare experiments and analyze how different model parameters affect model performance.

Follow the relevant steps below to compare experiments in H2O LLM Studio.

1. On the H2O LLM Studio left-navigation pane, click View experiments.

2. Click Compare experiments.

3. Select the experiments you want to compare.

4. Click Compare experiments.

   The Charts tab visually represents the comparison of train/validation loss, metrics, and learning rate of selected experiments. The Config tab compares the configuration settings of selected experiments.

:::info note
In addition, H2O LLM Studio also integrates with W&B, two powerful experiment tracking platforms. By enabling W&B logging when starting an experiment, you can easily track and visualize all aspects of your experiment in real time. This includes model performance, hyperparameter tuning, and other relevant metrics.
:::

File: documentation/docs/guide/experiments/create-an-experiment.md Copy Chapter

import Icon from "@mui/material/Icon"

Create an experiment

Follow the relevant steps below to create an experiment in H2O LLM Studio.

1. On the H2O LLM Studio left-navigation pane, click Create experiment. Alternatively, you can click New experiment on the more_vert Kebab menu of the View datasets page.

2. Select the Dataset you want to use to fine-tune an LLM model.

3. Select the Problem type.

4. Provide a meaningful Experiment name.

5. Define the parameters. The most important parameters are:

   • LLM Backbone: This parameter determines the LLM architecture to use. It is the foundation model that you continue training. H2O LLM Studio has a predefined list of recommended foundation models available in the dropdown list. You can also type in the name of a Hugging Face model that is not in the list, for example: h2oai/h2o-danube2-1.8b-sft or the path of a local folder that has the model you would like to fine-tune.
   • Mask Prompt Labels: This option controls whether to mask the prompt labels during training and only train on the loss of the answer.
   • Hyperparameters such as Learning rate, Batch size, and number of epochs determine the training process. You can refer to the tooltips that are shown next to each hyperparameter in the GUI to learn more about them.
   • Evaluate Before Training: This option lets you evaluate the model before training, which can help you judge the quality of the LLM backbone before fine-tuning.

   H2O LLM Studio provides several metric options for evaluating the performance of your model. In addition to the BLEU score, H2O LLM Studio also offers the GPT3.5 and GPT4 metrics that utilize the OpenAI API to determine whether the predicted answer is more favorable than the ground truth answer. To use these metrics, you can either export your OpenAI API key as an environment variable before starting LLM Studio, or you can specify it in the Settings menu within the UI.

   :::info note
   H2O LLM Studio provides an overview of all the parameters you need to specify for your experiment. The default settings are suitable when you first start an experiment. To learn more about the parameters, see Experiment settings.
   :::

6. Click Run experiment.

Run an experiment on the OASST data via CLI Copy Chapter

The steps below provide an example of how to to run an experiment on OASST data via the command line interface (CLI).

1. Get the training dataset (train_full.csv), OpenAssistant Conversations Dataset OASST2 and place it into the examples/data_oasst2 folder; or download it directly using the Kaggle API command given below.

kaggle kernels output philippsinger/openassistant-conversations-dataset-oasst2 -p examples/data_oasst2/

2. Go into the interactive shell or open a new terminal window. Install the dependencies first, if you have not installed them already.

make setup  # installs all dependencies
make shell

3. Run the following command to run the experiment.

python llm_studio/train.py -Y examples/example_oasst2.yaml

After the experiment is completed, you can find all output artifacts in the examples/output_oasst2 folder.
You can then use the prompt.py script to chat with your model.

python llm_studio/prompt.py -e examples/output_oasst2

4. To publish the model to Hugging Face, use the following command:

python llm_studio/publish_to_hugging_face.py -p {path_to_experiment} -d {device} -a {api_key} -u {user_id} -m {model_name} -s {safe_serialization}

File: documentation/docs/guide/experiments/evaluate-model-using-llm.md Copy Chapter

Evaluate model using an AI judge

H2O LLM Studio provides the option to use an AI Judge like ChatGPT or a local LLM deployment to evaluate a fine-tuned model.

Follow the instructions below to specify a local LLM to evaluate the responses of the fine-tuned model.

1. Have an endpoint running of the local LLM deployment, which supports the OpenAI API format; specifically the Chat Completions API.

2. Start the H2O LLM Studio server with the following environment variable that points to the endpoint.

   text Copy

   OPENAI_API_BASE="http://111.111.111.111:8000/v1"

3. Once H2O LLM Studio is up and running, click Settings on the left navigation panel to validate that the endpoint is being used correctly. The Use OpenAI API on Azure setting must be set to Off, and the environment variable that was set above should be the OpenAI API Endpoint value as shown below.

   :::info
   Note that changing the value of this field here on the GUI has no effect. This is only for testing the correct setting of the environment variable.
   :::

4. Run an experiment using GPT as the Metric and the relevant model name available at your endpoint as the Metric Gpt Model.

5. Validate that it is working as intended by checking the logs. Calls to the LLM judge should now be directed to your own LLM endpoint.