## File: README.md

Welcome to H2O LLM Studio, a framework and no-code GUI designed for fine-tuning state-of-the-art large language models (LLMs).

[](https://user-images.githubusercontent.com/1069138/233859311-32aa1f8c-4d68-47ac-8cd9-9313171ff9f9.png)[](https://user-images.githubusercontent.com/1069138/233859315-e6928aa7-28d2-420b-8366-bc7323c368ca.png) ## Jump to - [With H2O LLM Studio, you can](#with-h2o-llm-studio-you-can) - [Quickstart](#quickstart) - [What's New](#whats-new) - [Setup](#setup) - [Recommended Install](#recommended-install) - [Virtual Environments](#virtual-environments) - [Run H2O LLM Studio GUI](#run-h2o-llm-studio-gui) - [Run H2O LLM Studio GUI using Docker](#run-h2o-llm-studio-gui-using-docker) - [Run H2O LLM Studio with command line interface (CLI)](#run-h2o-llm-studio-with-command-line-interface-cli) - [Troubleshooting](#troubleshooting) - [Data format and example data](#data-format-and-example-data) - [Training your model](#training-your-model) - [Example: Run on OASST data via CLI](#example-run-on-oasst-data-via-cli) - [Model checkpoints](#model-checkpoints) - [Documentation](#documentation) - [Contributing](#contributing) - [License](#license) ## With H2O LLM Studio, you can - easily and effectively fine-tune LLMs **without the need for any coding experience**. - use a **graphical user interface (GUI)** specially designed for large language models. - fine-tune any LLM using a large variety of hyperparameters. - use recent fine-tuning techniques such as [Low-Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685) and 8-bit model training with a low memory footprint. - use Reinforcement Learning (RL) to fine-tune your model (experimental). - use advanced evaluation metrics to judge generated answers by the model. - track and compare your model performance visually. In addition, [W&B](https://wandb.ai/) integration can be used. - chat with your model and get instant feedback on your model performance. - easily export your model to the [Hugging Face Hub](https://huggingface.co/) and share it with the community. ## Quickstart For questions, discussing, or just hanging out, come and join our [Discord](https://discord.gg/WKhYMWcVbq)! Use cloud-based runpod.io instance to run the latest version of H2O LLM Studio with GUI. [](https://www.runpod.io/console/deploy?template=vf9ppiy56z) Using CLI for fine-tuning LLMs: [](https://www.kaggle.com/code/ilu000/h2o-llm-studio-cli/) [](https://colab.research.google.com/drive/1soqfJjwDJwjjH-VzZYO_pUeLx5xY4N1K?usp=sharing) ## What's New - [PR 788](https://github.com/h2oai/h2o-llmstudio/pull/788) New problem type for Causal Regression Modeling allows to train single target regression data using LLMs. - [PR 747](https://github.com/h2oai/h2o-llmstudio/pull/747) Fully removed RLHF in favor of DPO/IPO/KTO optimization. - [PR 741](https://github.com/h2oai/h2o-llmstudio/pull/741) Removing separate max length settings for prompt and answer in favor of a single `max_length` settings better resembling `chat_template` functionality from `transformers`. - [PR 592](https://github.com/h2oai/h2o-llmstudio/pull/599) Added `KTOPairLoss` for DPO modeling allowing to train models with simple preference data. Data currently needs to be manually prepared by randomly matching positive and negative examples as pairs. - [PR 592](https://github.com/h2oai/h2o-llmstudio/pull/592) Starting to deprecate RLHF in favor of DPO/IPO optimization. Training is disabled, but old experiments are still viewable. RLHF will be fully removed in a future release. - [PR 530](https://github.com/h2oai/h2o-llmstudio/pull/530) Introduced a new problem type for DPO/IPO optimization. This optimization technique can be used as an alternative to RLHF. - [PR 288](https://github.com/h2oai/h2o-llmstudio/pull/288) Introduced DeepSpeed for sharded training allowing to train larger models on machines with multiple GPUs. Requires NVLink. This feature replaces FSDP and offers more flexibility. DeepSpeed requires a system installation of CUDA Toolkit and we recommend using version 12.1. See [Recommended Install](#recommended-install). - [PR 449](https://github.com/h2oai/h2o-llmstudio/pull/449) New problem type for Causal Classification Modeling allows to train binary and multiclass models using LLMs. - [PR 364](https://github.com/h2oai/h2o-llmstudio/pull/364) User secrets are now handled more securely and flexible. Support for handling secrets using the 'keyring' library was added. User settings are tried to be migrated automatically. Please note that due to current rapid development we cannot guarantee full backwards compatibility of new functionality. We thus recommend to pin the version of the framework to the one you used for your experiments. For resetting, please delete/backup your `data` and `output` folders. ## Setup H2O LLM Studio requires a machine with Ubuntu 16.04+ and at least one recent NVIDIA GPU with NVIDIA drivers version >= 470.57.02. For larger models, we recommend at least 24GB of GPU memory. For more information about installation prerequisites, see the [Set up H2O LLM Studio](https://docs.h2o.ai/h2o-llmstudio/get-started/set-up-llm-studio#prerequisites) guide in the documentation. For a performance comparison of different GPUs, see the [H2O LLM Studio performance](https://h2oai.github.io/h2o-llmstudio/get-started/llm-studio-performance) guide in the documentation. ### Recommended Install The recommended way to install H2O LLM Studio is using `uv` with Python 3.10. To install Python 3.10 on Ubuntu 20.04+, execute the following commands: #### Installing NVIDIA Drivers (if required) If deploying on a 'bare metal' machine running Ubuntu, one may need to install the required NVIDIA drivers and CUDA. The following commands show how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an example. One can update the following based on their OS. ```bash wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb sudo dpkg -i cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb sudo cp /var/cuda-repo-ubuntu2204-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/ sudo apt-get update sudo apt-get -y install cuda-toolkit-12-4 ``` ### Virtual environments We offer various ways of setting up the necessary python environment. #### UV virtual environment The following command will create a virtual environment using `uv` and will install the dependencies: ```bash make setup ``` ## Run H2O LLM Studio GUI You can start H2O LLM Studio using the following command: ```bash make llmstudio ``` This command will start the [H2O Wave](https://github.com/h2oai/wave) server and app. Navigate to (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models! If you are running H2O LLM Studio with a custom environment other than `uv`, you need to start the app as follows: ```bash H2O_WAVE_MAX_REQUEST_SIZE=25MB \ H2O_WAVE_NO_LOG=true \ H2O_WAVE_PRIVATE_DIR="/download/@output/download" \ wave run llm_studio.app ``` ## Run H2O LLM Studio GUI using Docker Install Docker first by following instructions from [NVIDIA Containers](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker). Make sure to have `nvidia-container-toolkit` installed on your machine as outlined in the instructions. H2O LLM Studio images are stored in the h2oai Docker Hub container repository. ```bash mkdir -p `pwd`/llmstudio_mnt chmod 777 `pwd`/llmstudio_mnt # make sure to pull latest image if you still have a prior version cached docker pull h2oairelease/h2oai-llmstudio-app:latest # run the container docker run \ --runtime=nvidia \ --shm-size=64g \ --init \ --rm \ -it \ -u `id -u`:`id -g` \ -p 10101:10101 \ -v `pwd`/llmstudio_mnt:/mount \ h2oairelease/h2oai-llmstudio-app:latest ``` Navigate to (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models! (Note other helpful docker commands are `docker ps` and `docker kill`.) ## Run H2O LLM Studio with command line interface (CLI) You can also use H2O LLM Studio with the command line interface (CLI) and specify the configuration .yaml file that contains all the experiment parameters. To fine-tune using H2O LLM Studio with CLI use the following command: ```bash uv run python llm_studio/train.py -Y {path_to_config_yaml_file} ``` To run on multiple GPUs in DDP mode, run the following command: ```bash bash distributed_train.sh {NR_OF_GPUS} -Y {path_to_config_yaml_file} ``` By default, the framework will run on the first `k` GPUs. If you want to specify specific GPUs to run on, use the `CUDA_VISIBLE_DEVICES` environment variable before the command. To start an interactive chat with your trained model, use the following command: ```bash uv run python llm_studio/prompt.py -e {experiment_name} ``` where `experiment_name` is the output folder of the experiment you want to chat with (see configuration). The interactive chat will also work with model that were fine-tuned using the UI. To publish the model to Hugging Face, use the following command: ```bash uv run python llm_studio/publish_to_hugging_face.py -p {path_to_experiment} -d {device} -a {api_key} -u {user_id} -m {model_name} -s {safe_serialization} ``` `path_to_experiment` is the output folder of the experiment. `device` is the target device for running the model, either 'cpu' or 'cuda:0'. Default is 'cuda:0'. `api_key` is the Hugging Face API Key. If the user is logged in, it can be omitted. `user_id` is the Hugging Face user ID. If the user is logged in, it can be omitted. `model_name` is the name of the model to be published on Hugging Face. It can be omitted. `safe_serialization` is a flag indicating whether safe serialization should be used. Default is True. ## Troubleshooting If running on cloud-based machines such as runpod, you may need to set the following environment variable to allow the H2O Wave server to accept connections from the proxy: ```bash H2O_WAVE_ALLOWED_ORIGINS="*" ``` If you are experiencing timeouts when running the H2O Wave server remotely, you can increase the timeout by setting the following environment variables: ```bash H2O_WAVE_APP_CONNECT_TIMEOUT="15" H2O_WAVE_APP_WRITE_TIMEOUT="15" H2O_WAVE_APP_READ_TIMEOUT="15" H2O_WAVE_APP_POOL_TIMEOUT="15" ``` All default to 5 (seconds). Increase them if you are experiencing timeouts. Use -1 to disable the timeout. ## Data format and example data For details on the data format required when importing your data or example data that you can use to try out H2O LLM Studio, see [Data format](https://docs.h2o.ai/h2o-llmstudio/guide/datasets/data-connectors-format#data-format) in the H2O LLM Studio documentation. ## Training your model With H2O LLM Studio, training your large language model is easy and intuitive. First, upload your dataset and then start training your model. Start by [creating an experiment](https://docs.h2o.ai/h2o-llmstudio/guide/experiments/create-an-experiment). You can then [monitor and manage your experiment](https://docs.h2o.ai/h2o-llmstudio/guide/experiments/view-an-experiment), [compare experiments](https://docs.h2o.ai/h2o-llmstudio/guide/experiments/compare-experiments), or [push the model to Hugging Face](https://docs.h2o.ai/h2o-llmstudio/guide/experiments/export-trained-model) to share it with the community. ## Example: Run on OASST data via CLI As an example, you can run an experiment on the OASST data via CLI. For instructions, see [Run an experiment on the OASST data](https://docs.h2o.ai/h2o-llmstudio/guide/experiments/create-an-experiment#run-an-experiment-on-the-oasst-data-via-cli) guide in the H2O LLM Studio documentation. ## Model checkpoints All open-source datasets and models are posted on [H2O.ai's Hugging Face page](https://huggingface.co/h2oai/) and our [H2OGPT](https://github.com/h2oai/h2ogpt) repository. ## Documentation Detailed documentation and frequently asked questions (FAQs) for H2O LLM Studio can be found at . If you wish to contribute to the docs, navigate to the `/documentation` folder of this repo and refer to the [README.md](documentation/README.md) for more information. ## Contributing We are happy to accept contributions to the H2O LLM Studio project. Please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. ## License H2O LLM Studio is licensed under the Apache 2.0 license. Please see the [LICENSE](LICENSE) file for more information. --- ## File: documentation/docs/get-started/core-features.md --- --- # Core features ## No-code fine-tuning NLP practitioners can easily fine-tune models without the need for code expertise. The user interface, which is specifically designed for LLMs, allows users to upload large datasets easily and configure [hyperparameters](../concepts#parameters-and-hyperparameters) to fine-tune the model. ## Highly customizable (wide range of hyperparameters) H2O LLM Studio supports a wide variety of hyperparameters that can be used to fine-tune the model and supports the following fine-tuning techniques to enable advanced customization: - [Low-Rank Adaptation (LoRA)](../concepts#lora-low-rank-adaptation) - [8-bit model training with a low memory footprint](../concepts#8-bit-model-training-with-a-low-memory-footprint) ## Advanced evaluation metrics and experiment comparison Advanced evaluation metrics in H2O LLM Studio can be used to validate the answers generated by the LLM. This helps to make data-driven decisions about the model. It also offers visual tracking and comparison of experiment performance, making it easy to analyze and compare different fine-tuned models.You can also visualize how different parameters affect the model performance, and optionally use the [W&B](https://wandb.ai/) integration to track and log your experiments. ## Instant publishing models H2O LLM Studio enables easy model sharing with the community by allowing you to export the model to the [Hugging Face Hub](https://huggingface.co/h2oai) with a single click. ## Instant feedback on model performance Additionally, H2O LLM Studio lets you chat with the fine-tuned model and receive instant feedback about model performance. --- ## File: documentation/docs/get-started/llm-studio-flow.md --- --- # Model flow The flow of creating and fine-tuning large language models using H2O LLM Studio can be summarized in the following sequential steps: - [Step 1: Import a dataset](#step-1-import-a-dataset) - [Step 2: Create an experiment](#step-2-create-an-experiment) - [Step 3: Monitor an experiment](#step-3-monitor-an-experiment) - [Step 4: Compare experiments](#step-4-compare-experiments) - [Step 5: Export a model to Hugging Face Hub](#step-5-export-a-model-to-hugging-face-hub) ## Step 1: Import a dataset As the first step in the experiment flow, prep your data and import your dataset to H2O LLM Studio. - To learn about supported data connectors and data format, see [Supported data connectors and format](../guide/datasets/data-connectors-format). - To learn about how to import a dataset to H2O LLM Studio, see [Import a dataset](../guide/datasets/import-dataset). - To learn about reviewing and editing a dataset, see [View and manage dataset](../guide/datasets/view-dataset.md). ## Step 2: Create an experiment As the second step in the experiment flow, create an experiment using the imported dataset. H2O LLM Studio offers several hyperparameter settings that you can adjust for your experiment model. To ensure that your training process is effective, you may need to specify the [hyperparameters](../concepts#parameters-and-hyperparameters) like learning rate, batch size, and the number of epochs. H2O LLM Studio provides an overview of all the parameters you’ll need to specify for your experiment. - To learn about creating a new experiment, see [Create an experiment](../guide/experiments/create-an-experiment.md). - To learn about the settings available for creating an experiment, see [Experiment settings](../guide/experiments/experiment-settings.md). ## Step 3: Monitor an experiment As the third step in the experiment flow, monitor the launched experiment. H2O LLM Studio allows you to inspect your experiment (model) during and after model training. Simple interactive graphs in H2O LLM Studio allow you to understand the impact of selected hyperparameter values during and after model training. You can then adjust the [hyperparameters](../concepts#parameters-and-hyperparameters) to further optimize model performance. To learn about viewing and monitoring an experiment, see [View and manage experiments](../guide/experiments/view-an-experiment.md). ## Step 4: Compare experiments The H2O LLM studio provides a useful feature that allows comparing various experiments and analyzing how different model parameters affect model performance. This feature is a powerful tool for fine-tuning your machine-learning models and ensuring they meet your desired performance metrics. To learn about comparing multiple experiments, see [Compare experiments](../guide/experiments/compare-experiments.md). ## Step 5: Export a model to Hugging Face Hub As the final step in the experiment flow, you can export the fine-tuned model to Hugging Face with a single click. To learn about exporting a trained model to Hugging Face Hub, see, [Export trained model to Hugging Face](../guide/experiments/export-trained-model.md). --- ## File: documentation/docs/get-started/llm-studio-performance.md --- --- # H2O LLM Studio performance Setting up and running H2O LLM Studio requires the following minimal [prerequisites](set-up-llm-studio.md#prerequisites). This page lists out the speed and performance metrics of H2O LLM Studio based on different hardware setups. The following metrics were measured. - **Hardware setup:** The type and number of computing devices used to train the model. - **LLM backbone:** The underlying architecture of the language model. For more information, see [LLM backbone](concepts.md#llm-backbone). - **Quantization:** A technique used to reduce the size and memory requirements of the model. For more information, see [Quantization](concepts.md#quantization). - **Train**: The amount of time it took to train the model in hours and minutes. - **Validation:** The amount of time it took to validate the mode in hours and minutes. | Hardware setup | LLM backbone | Quantization | Train (hh:mm:ss)| Validation (hh:mm:ss) | |---|---|---|---|---| | 8xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 11:35 | 3:32 | | 4xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 21:13 | 06:35 | | 2xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 37:04 | 12:21 | | 1xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 1:25:29 | 15:50 | | 8xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 14:26 | 06:13 | | 4xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 26:55 | 11:59 | | 2xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 48:24 | 23:37 | | 1xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 1:26:59 | 42:17 | | 8xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 4xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 2xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 1xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 8xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 25:07 | 10:58 | | 4xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 48:43 | 21:25 | | 2xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 1:30:45 | 42:06 | | 1xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 2:44:36 | 1:14:20 | | 8xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | | 4xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | | 2xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | | 1xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | |---|---|---|---|---| | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 7:04 | 3:55 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 13:14 | 7:23 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 23:36 | 13:25 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 9:44 | 6:30 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 18:34 | 12:16 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 34:06 | 21:51 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 11:46 | 5:56 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 21:54 | 11:17 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 39:10 | 18:55 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 16:51 | 10:35 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 32:05 | 21:00 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 59:11 | 36:53 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 1:13:33 | 46:02 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 2:20:44 | 1:33:42 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 4:23:57 | 2:44:51 | :::info The runtimes were gathered using the default parameters. Expand to see the default parameters ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ::: --- ## File: documentation/docs/get-started/set-up-llm-studio.md --- --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; # Set up H2O LLM Studio This page guides you through setting up and installing H2O LLM Studio on your local system. First, download the H2O LLM Studio package from the [H2O LLM Studio Github repository](https://github.com/h2oai/h2o-llmstudio). You can use `git clone` or navigate to the [releases page](https://github.com/h2oai/h2o-llmstudio/releases) and download the `.zip` file found within the **Assets** of the relevant release. ## Prerequisites H2O LLM Studio requires the following minimum requirements: - A machine with Ubuntu 16.04+ with atleast one recent Nvidia GPU - Have at least 128GB+ of system RAM. Larger models and complex tasks may require 256GB+ or more. - Nvidia drivers v470.57.02 or a later version - Access to the following URLs: - developer.download.nvidia.com - pypi.org - huggingface.co - download.pytorch.org - cdn-lfs.huggingface.co :::info Notes - Atleast 24GB of GPU memory is recommended for larger models. - For more information on performance benchmarks based on the hardware setup, see [H2O LLM Studio performance](llm-studio-performance.md). - The required URLs are accessible by default when you start a GCP instance, however, if you have network rules or custom firewalls in place, it is recommended to confirm that the URLs are accessible before running `make setup`. ::: ## Installation :::note Installation methods The recommended way to install H2O LLM Studio is using `uv` with Python 3.10. To install Python 3.10 on Ubuntu 16.04+, execute the following commands. **Install NVIDIA drivers (if required)** If you are deploying on a 'bare metal' machine running Ubuntu, you may need to install the required Nvidia drivers and CUDA. The following commands show how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an example. You can update the following based on your respective operating system. wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin{" "} sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 wget https://developer.download.nvidia.com/compute/cuda/11.4.3/local_installers/cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb{" "} sudo dpkg -i cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb sudo apt-key add /var/cuda-repo-ubuntu2004-11-4-local/7fa2af80.pub sudo apt-get -y update sudo apt-get -y install cuda **Create virtual environment (uv) ** The following command creates a virtual environment using `uv` and will install the dependencies. `make setup` Follow the steps below to install H2O LLM Studio on a Windows machine using Windows Subsystem for Linux{" "} [WSL2](https://learn.microsoft.com/en-us/windows/wsl/) 1. Download the{" "} latest nvidia driver {" "} for Windows.{" "} 2. Open PowerShell or a Windows Command Prompt window in administrator mode.{" "} 3. Run the following command to confirm that the driver is installed properly and see the driver version. `nvidia-smi` 4. Run the following command to install WSL2. `wsl --install` 5. Launch the WSL2 Ubuntu installation. 6. Install the{" "} WSL2 Nvidia Cuda Drivers . wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin{" "} sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb{" "} sudo dpkg -i cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb sudo cp /var/cuda-repo-wsl-ubuntu-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/ sudo apt-get update sudo apt-get -y install cuda 7. Set up the required python system installs (Python 3.10). sudo add-apt-repository ppa:deadsnakes/ppa sudo apt install python3.10 sudo apt-get install python3.10-distutils curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10 8. Create the virtual environment. sudo apt install -y python3.10-venv python3 -m venv llmstudio source llmstudio/bin/activate 9.Clone the H2O LLM Studio repository locally. git clone https://github.com/h2oai/h2o-llmstudio.git cd h2o-llmstudio 10. Install H2O LLM Studio dependencies. `make setup` 11. Run the H2O LLM Studio application. H2O_WAVE_MAX_REQUEST_SIZE=25MB \ H2O_WAVE_NO_LOG=True \ H2O_WAVE_PRIVATE_DIR="/download/@output/download" \ wave run llm_studio.app This will start the H2O Wave server and the H2O LLM Studio app. Navigate to http://localhost:10101/ (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models. ::: ## Install custom package If required, you can install additional Python packages into your environment. You can directly install the custom package by running the following command. ```bash uv add package_name ``` ## Run H2O LLM Studio There are several ways to run H2O LLM Studio depending on your requirements. 1. [Run H2O LLM Studio GUI](#run-h2o-llm-studio-gui) 2. [Run using Docker](#run-using-docker) 3. [Run by building your own Docker image](#run-by-building-your-own-docker-image) 4. [Run with the CLI (command-line interface)](#run-with-command-line-interface-cli) ### Run H2O LLM Studio GUI Run the following command to start the H2O LLM Studio. ```sh make llmstudio ``` This will start the H2O Wave server and the H2O LLM Studio app. Navigate to [http://localhost:10101/](http://localhost:10101/) (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models. If you are running H2O LLM Studio with a custom environment other than `uv`, start the app as follows: ```sh H2O_WAVE_MAX_REQUEST_SIZE=25MB \ H2O_WAVE_NO_LOG=True \ H2O_WAVE_PRIVATE_DIR="/download/@output/download" \ wave run llm_studio.app ``` ### Run using Docker First, install Docker by following the instructions from the [NVIDIA Container Installation Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker). H2O LLM Studio images are stored in the `h2oai dockerhub` container repository. ```sh mkdir -p `pwd`/llmstudio_mnt docker run \ --runtime=nvidia \ --shm-size=64g \ --init \ --rm \ -it \ -p 10101:10101 \ -v `pwd`/llmstudio_mnt:/mount \ h2oairelease/h2oai-llmstudio-app:latest ``` Navigate to [http://localhost:10101/](http://localhost:10101/) (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models. :::info Other helpful docker commands are `docker ps` and `docker kill`. ::: ### Run by building your own Docker image ```sh docker build -t h2o-llmstudio . mkdir -p `pwd`/llmstudio_mnt docker run \ --runtime=nvidia \ --shm-size=64g \ --init \ --rm \ -it \ -p 10101:10101 \ -v `pwd`/llmstudio_mnt:/mount \ h2o-llmstudio ``` ### Run with command line interface (CLI) You can also use H2O LLM Studio with the command line interface (CLI) and specify the configuration .yaml file that contains all the experiment parameters. To finetune using H2O LLM Studio with CLI and to specify the path to the configuration file that contains the experiment parameters, run: ```sh uv run python llm_studio/train.py -Y {path_to_config_yaml_file} ``` To run on multiple GPUs in DDP mode, run: ```sh bash distributed_train.sh {NR_OF_GPUS} -Y {path_to_config_yaml_file} ``` :::info By default, the framework will run on the first `k` GPUs. If you want to specify specific GPUs to run on, use the `CUDA_VISIBLE_DEVICES` environment variable before the command. ::: To start an interactive chat with your trained model, run: ```sh uv run python llm_studio/prompt.py -e {experiment_name} ``` `experiment_name` is the output folder of the experiment you want to chat with. The interactive chat will also work with models that were fine-tuned using the GUI. --- ## File: documentation/docs/get-started/videos.md --- --- import ReactPlayer from 'react-player' # Videos ## Discovering the Potential of LLMs :::info Note In this video, Andreea Turcu delves in-depth into the world of language models, showcasing how users can use H2O.ai's LLM Studio to their full advantage. ::: --- ## The Fine Art of Fine-Tuning Large Language Models :::info Note In this video, Pascal Pfeiffer, Principal Data Scientist at H2O.ai and Kaggle Grandmaster, announces the release of H2O LLM Studio and talks about fine-tuning LLMs using H2O LLM Studio at H2O World India 2023. ::: --- ## Basic introduction to H2O LLM Studio :::info Note In this video, Avkash Chauhan, founder of Prodramp Inc, gives a basic introduction about H2O LLM Studio. ::: ---- ## LLM Fine-Tuning, Falcon 40b, and the State of Open-Source :::info Note In this video, Pascal Pfeiffer, the Principal Data Scientist at h2o.ai is interviewed about LLM fine-tuning, being a Kaggle Grandmaster, H2O.ai, Falcon 40b, the state of open-source, and more. ::: --- ## File: documentation/docs/get-started/what-is-h2o-llm-studio.md --- --- # What is H2O LLM Studio? H2O LLM Studio is an open-source, no-code [LLM](../concepts#llm) graphical user interface (GUI) designed for fine-tuning state-of-the-art large language models. [Fine-tuning](../concepts#fine-tuning) a pretrained language model requires coding expertise and extensive knowledge about the model and its [hyperparameters](../concepts#parameters-and-hyperparameters), however H2O LLM Studio enables NLP practitioners to fine-tune their LLMs easily with no need for coding and better flexibility over customization. H2O LLM Studio also lets you chat with the fine-tuned model and receive instant feedback about model performance. ## Who is H2O LLM Studio for? H2O LLM Studio is a free and open-source tool that is designed for anyone who wants to fine-tune their own language models. It is designed to be easy to use and accessible to everyone regardless of their technical expertise. NLP practitioners and data scientists in particular may find it useful to easily and effectively create and fine-tune large language models. --- ## File: documentation/docs/guide/experiments/compare-experiments.md --- --- # Compare experiments Using H2O LLM Studio, you can compare experiments and analyze how different model parameters affect model performance. Follow the relevant steps below to compare experiments in H2O LLM Studio. 1. On the H2O LLM Studio left-navigation pane, click **View experiments**. 2. Click **Compare experiments**. 3. Select the experiments you want to compare. 4. Click **Compare experiments**. The **Charts** tab visually represents the comparison of train/validation loss, metrics, and learning rate of selected experiments. The **Config** tab compares the configuration settings of selected experiments. :::info note In addition, H2O LLM Studio also integrates with [W&B](https://wandb.ai/), two powerful experiment tracking platforms. By enabling W&B logging when starting an experiment, you can easily track and visualize all aspects of your experiment in real time. This includes model performance, hyperparameter tuning, and other relevant metrics. ::: --- ## File: documentation/docs/guide/experiments/create-an-experiment.md --- --- import Icon from "@mui/material/Icon" # Create an experiment Follow the relevant steps below to create an experiment in H2O LLM Studio. 1. On the H2O LLM Studio left-navigation pane, click **Create experiment**. Alternatively, you can click **New experiment** on the more_vert Kebab menu of the [View datasets](../datasets/view-dataset.md) page. 2. Select the **Dataset** you want to use to fine-tune an LLM model. 3. Select the **Problem type**. 4. Provide a meaningful **Experiment name**. 5. Define the parameters. The most important parameters are: - **LLM Backbone**: This parameter determines the LLM architecture to use. It is the foundation model that you continue training. H2O LLM Studio has a predefined list of recommended foundation models available in the dropdown list. You can also type in the name of a [Hugging Face model](https://huggingface.co/models) that is not in the list, for example: `h2oai/h2o-danube2-1.8b-sft` or the path of a local folder that has the model you would like to fine-tune. - **Mask Prompt Labels**: This option controls whether to mask the prompt labels during training and only train on the loss of the answer. - Hyperparameters such as **Learning rate**, **Batch size**, and number of epochs determine the training process. You can refer to the tooltips that are shown next to each hyperparameter in the GUI to learn more about them. - **Evaluate Before Training**: This option lets you evaluate the model before training, which can help you judge the quality of the LLM backbone before fine-tuning. H2O LLM Studio provides several metric options for evaluating the performance of your model. In addition to the BLEU score, H2O LLM Studio also offers the GPT3.5 and GPT4 metrics that utilize the OpenAI API to determine whether the predicted answer is more favorable than the ground truth answer. To use these metrics, you can either export your OpenAI API key as an environment variable before starting LLM Studio, or you can specify it in the **Settings** menu within the UI. :::info note H2O LLM Studio provides an overview of all the parameters you need to specify for your experiment. The default settings are suitable when you first start an experiment. To learn more about the parameters, see [Experiment settings](experiment-settings.md). ::: 6. Click **Run experiment**. ## Run an experiment on the OASST data via CLI The steps below provide an example of how to to run an experiment on [OASST](https://huggingface.co/OpenAssistant) data via the command line interface (CLI). 1. Get the training dataset (`train_full.csv`), [OpenAssistant Conversations Dataset OASST2](https://www.kaggle.com/code/philippsinger/openassistant-conversations-dataset-oasst2?scriptVersionId=160485459) and place it into the `examples/data_oasst2` folder; or download it directly using the [Kaggle API](https://www.kaggle.com/docs/api) command given below. ```bash kaggle kernels output philippsinger/openassistant-conversations-dataset-oasst2 -p examples/data_oasst2/ ``` 2. Go into the interactive shell or open a new terminal window. Install the dependencies first, if you have not installed them already. ```bash make setup # installs all dependencies make shell ``` 3. Run the following command to run the experiment. ```bash python llm_studio/train.py -Y examples/example_oasst2.yaml ``` After the experiment is completed, you can find all output artifacts in the `examples/output_oasst2` folder. You can then use the `prompt.py` script to chat with your model. ```bash python llm_studio/prompt.py -e examples/output_oasst2 ``` 4. To publish the model to Hugging Face, use the following command: ```bash python llm_studio/publish_to_hugging_face.py -p {path_to_experiment} -d {device} -a {api_key} -u {user_id} -m {model_name} -s {safe_serialization} ``` --- ## File: documentation/docs/guide/experiments/evaluate-model-using-llm.md # Evaluate model using an AI judge H2O LLM Studio provides the option to use an AI Judge like ChatGPT or a local LLM deployment to evaluate a fine-tuned model. Follow the instructions below to specify a local LLM to evaluate the responses of the fine-tuned model. 1. Have an endpoint running of the local LLM deployment, which supports the OpenAI API format; specifically the [Chat Completions API](https://platform.openai.com/docs/guides/text-generation/chat-completions-api). 2. Start the H2O LLM Studio server with the following environment variable that points to the endpoint. ``` OPENAI_API_BASE="http://111.111.111.111:8000/v1" ``` 3. Once H2O LLM Studio is up and running, click **Settings** on the left navigation panel to validate that the endpoint is being used correctly. The **Use OpenAI API on Azure** setting must be set to Off, and the environment variable that was set above should be the **OpenAI API Endpoint** value as shown below. :::info Note that changing the value of this field here on the GUI has no effect. This is only for testing the correct setting of the environment variable. ::: 4. Run an experiment using `GPT` as the **Metric** and the relevant model name available at your endpoint as the **Metric Gpt Model**. 5. Validate that it is working as intended by checking the logs. Calls to the LLM judge should now be directed to your own LLM endpoint.