{"owner":"NVIDIA","repo":"Megatron-LM","hasSkills":true,"totalSkillsCount":3,"totalTokensCount":4785,"categories":["plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["tools/openwebtext/README.md","tools/retro/README.md","tools/retro/sft/README.md"],"skills":{"tools/openwebtext/README.md":"The following steps show how to prepare training dataset to train the mode.\n\n# Libraries to install\n\n```\n    pip install ftfy langdetect numpy torch pandas nltk sentencepiece boto3 tqdm regex bs4 newspaper3k htmlmin tldextract \n    git clone https://github.com/mattilyra/LSH\n    cd LSH\n    python setup.py install\n``` \n\n# Download the dataset\n\n1. Download the deduplicated URLs from [jcpeterson](https://mega.nz/#F!EZZD0YwJ!9_PlEQzdMVLaNdKv_ICNVQ!cc4RgQQZ)\n2. Remove blacklisted URLs.\n```\npython blacklist_urls.py <path to the downloaded deduplicated URLs> <filename for clean urls. e.g. clean_urls.txt>\n```\n3. Download the content from the clean urls with [openwebtext's utilities](https://github.com/eukaryote31/openwebtext/blob/master/download.py). \n\n4. Merge the contents into one loose json file with 1 json per newline of the format `{'text': text, 'url': unique_url}`. It is important for the url to be unique.\n\n# Prepare the data for GPT training:\n\n1. Perform ftfy, english detection and remove documents with less than 128 tokens. This step can be sharded and run on shards.\n```\npython cleanup_dataset.py <input data file> <output cleaned data filename>\n```\nAdditional cleanup (e.g. remove documents less than 512 characters or dataset specific cleaning like stories, realnews datasets) can be done using `cleanup_fix_dataset.py`. More details can be found by running `python cleanup_fix_dataset.py --help`.\n2. Using LSH, find possible duplicates and store then in a file for later processing. The code supports saving and loading fingerprints for recurrent deduplications, and is also multithreaded for faster processing. More details are can be found by `python find_duplicate.py --help`.\n```\npython find_duplicates.py --inputs <pairlist list of input cleaned data files and keys, e.g. cc.json cc_id news.json news_id> --output <output possible duplicate urls filename>\n```\n3. Based on similarity measure defind inside function `is_similar` (default: 0.9), group urls that are similar. Basically, for each group, only one url we should keep and remove the rest.\n```\npython group_duplicate_urls.py <possible duplicate urls file> <output file containing similar urls>\n```\n4. Remove similar documents that were detected in the last step.\n```\npython remove_group_duplicates.py <file containing similar documents> <cleaned data file> <outputfile containing deduplicate data>\n```\n\n5. Shuffle the dataset.\n```\nshuf <cleaned deduped data file> -o train_data.json\n```\n\n# Deduplicating ngrams\n\nTo deduplicate the downstream tasks (e.g. lambada, squad) from the training dataset, we run the following command.\n\n```\npython filter_ngrams.py --tasks <name of the task, e.g. lambada, squad> --dedup-dataset <training dataset to deduplicate> <json key> --output <output training dataset>\n```\nWe use 13-grams by default for the deduplication. When we find a 13-gram match in a training document, we split the document into two pieces and remove the 13-gram along with 200 characters from the both side of the 13-gram. We also remove any splitted document with less than 200 characters or if a document got splitted more than 10 times. These parameters can be changed using corresponding arguments.\n\nOnly for the lambada task, we need to provide the path, `--lambada-path <path of the lambada test data>`.\n\nSeveral other features (e.g. save and load dictionary) have been added, look at `python filter_ngrams.py --help` for details.\n","tools/retro/README.md":"# Retro and InstructRetro\n\nRetro [(Borgeaud et al., 2022)](https://arxiv.org/abs/2112.04426) is an autoregressive decoder-only language model (LM)\npretrained with retrieval-augmentation.\nRetro features practical scalability to support large-scale pretraining from scratch by retrieving from trillions of\ntokens.\nPretraining with retrieval provides a more efficient storage mechanism of factual knowledge, when compared to storing\nfactual knowledge implicitly within the network's parameters, thus largely reducing model parameters while achieving\nlower perplexity than standard GPT.\nRetro also provides the flexibility to update the\nknowledge stored in LMs [(Wang et al., 2023a)](https://arxiv.org/abs/2304.06762)\nby updating the retrieval database without training LMs again.\n\nInstructRetro [(Wang et al., 2023b)](https://arxiv.org/abs/2310.07713) further scales up the size of Retro to 48B,\nfeaturing the largest LLM pretrained with retrieval (as of December 2023).\nThe obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity.\nWith instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on\ndownstream tasks in the zero-shot setting. Specifically, the average improvement of InstructRetro is 7% over its GPT\ncounterpart across 8 short-form QA tasks, 10% over GPT across 4 challenging long-form QA tasks, and 16% over GPT across\n3 summarization tasks. We also find that one can ablate the encoder from InstructRetro architecture and directly use the\nInstructRetro decoder backbone as GPT, while achieving comparable results.\n\nThis README provides an end-to-end tutorial to reproduce Retro and InstructRetro.\n\n# Contents\n\n* [Checkpoints](#checkpoints)\n* [End-to-end Reproduction Guide](#end-to-end-reproduction-guide)\n    * [Step 0: Prepare the environment](#step-0-prepare-the-environment)\n        * [Docker image](#docker-image)\n        * [Install dependencies](#install-dependencies)\n    * [Step 1: Build retrieval database](#step-1-build-retrieval-database)\n    * [Step 2: Pretraining](#step-2-pretraining)\n    * [Step 3: Perplexity evaluation](#step-3-perplexity-evaluation)\n    * [Step 4: Instruction tuning](#step-4-instruction-tuning)\n    * [Step 5: Downstream task evaluation](#step-5-downstream-task-evaluation)\n* [Citations](#citations)\n\n# Checkpoints\n\nWe provide the pretrained checkpoints of Retro and InstructRetro in the following table. The checkpoints are available\nto download through the following links:\n\n| Model                   | Size | Instruction Tuning | Download Link 1                                                    | Download Link 2                                                                | Download Link 3                                                                                      |\n|-------------------------|------|--------------------|--------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|\n| `retro-8b-base-4k`      | 8b   |                    | [Huggingface](https://huggingface.co/nvidia/retro-8b-base-4k)      | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-base-4k)      | [Google Drive](https://drive.google.com/drive/folders/1uSQ5DAsuvx_8XcbtnVfs_MGvEOcx0uK_?usp=sharing) |\n| `retro-8b-instruct-4k`  | 8b   | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-8b-instruct-4k)  | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-instruct-4k)  | [Google Drive](https://drive.google.com/drive/folders/1v5dKaSN0cm2lwyAWpFaJtlTrLhtMZXsI?usp=sharing) |\n| `retro-48b-base-4k`     | 48b  |                    | [Huggingface](https://huggingface.co/nvidia/retro-48b-base-4k)     | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-base-4k)     | [Google Drive](https://drive.google.com/drive/folders/1rtNpf0CiLElSHQcr3aLI3zgfI3teGTP5?usp=sharing) |\n| `retro-48b-instruct-4k` | 48b  | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-48b-instruct-4k) | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-instruct-4k) | [Google Drive](https://drive.google.com/drive/folders/1qdb0AQjSsAPGlWaIu3wgHPjf_nwLeY5h?usp=sharing) |\n\n# End-to-end Reproduction Guide\n\nIn this README, we provide an end-to-end reproduction guide for InstructRetro, covering from large-scale retrieval\nconstruction, pretraining, perplexity evaluation, instruction tuning, to downstream task evaluation.\n\nIf you are interested in evaluation only, we also [open-sourced our checkpoints](#checkpoints) and you can directly go\nto [Step 5](#step-5-downstream-task-evaluation) to evaluate the checkpoints on downstream tasks.\n\n## Step 0: Prepare the environment\n\nWe recommend using docker environment to run the code.\n\n### Docker image\n\nWe provide a docker build file in [tools/retro/examples/Dockerfile](examples/Dockerfile) for the reproduction. The\ndocker image is based on the [NGC docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) `nvcr.io/nvidia/pytorch:23.09-py3`.\n\n### Install dependencies\n\nClone the Megatron repo:\n\n```bash\ngit clone --branch InstructRetro https://github.com/NVIDIA/Megatron-LM.git\n```\n\nIf docker is not available, we recommend starting from a clean conda environment with the following runtime\ndependencies:\n\n- Python 3.10\n- NVIDIA CUDA® 12.2.1\n- NVIDIA cuBLAS 12.2.5.6\n- NVIDIA cuDNN 8.9.5\n- NVIDIA NCCL 2.18.5\n- PyTorch 2.1.0a0+32f93b1\n\nThen install Retro-specific dependencies, including:\n\n```bash\npip install -U faiss-gpu\npip install -U transformers\npip install -U sentencepiece\npip install -U h5py\npip install -U nltk\npip install -U einops\n```\n\n## Step 1: Build retrieval database\n\nIn this step, we build a large-scale retrieval database for InstructRetro\nthrough [Faiss](https://github.com/facebookresearch/faiss) to retrieve from trillions of tokens, and preprocess (and\nsave) the retrieval neighbors for the pretraining step.\n\nPlease refer to [tools/retro/build_db.md](build_db.md) for more details.\n\n## Step 2: Pretraining\n\n*Please strictly follow Step 1 to build the retrieval database before pretraining to make sure the preprocessed\nretrieval neighbors match the pretraining corpus.*\n\nIn the pretraining step, we support both pretraining from scratch and continued pretraining from a pretrained GPT model.\n\nWe provide a template pretraining script to pretrain 843M Retro from scratch. Prepare your own arguments and update our\ntemplates in [tools/retro/examples/pretrain_model.sh](examples/pretrain_model.sh). Please note that the data path should\nbe exactly matching the one used in Step 1 to make sure the preprocessed retrieval neighbors match the pretraining\ncorpus.\n\n[//]: # (Take the example of the Wikipedia corpus)\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\nAfter pretraining, the model checkpoints will be saved in the `--save` directory if you specified the arg\nin `pretrain_model.sh`.\n\nTo continue pretraining with retrieval from a pretrained GPT model, please specify `--load` in `pretrain_model.sh` to\nload the pretrained GPT model checkpoint (the architecture of GPT, including hidden size, number of layers, and\nactivation methods, should be exactly the same as the one used for Retro). You should also\nspecify  `--no-load-optim --finetune` to make sure the optimizer state is not loaded from the pretrained GPT model and\nthe continued pretraining with retrieval is from a clean start. After the first job / the first run, you will continue\npretraining with retrieval from your last checkpoint. In the follow-up jobs, you should launch the pretraining without\nthe flags `--no-load-optim --finetune` to make sure the optimizer state is correctly loaded from your last job.\n\n## Step 3: Perplexity evaluation\n\nDuring pretraining, we will automatically evaluate the model perplexity on the specified validation corpus\nevery `--eval-interval` steps. The validation corpus should be exactly the same as the one used in Step 1 to make sure\nthe preprocessed retrieval neighbors match the pretraining corpus.\n\nTo evaluate the perplexity of a pretrained model, please add `--skip-train` in `pretrain_model.sh` to skip the\npretraining step and only evaluate the perplexity of the model specified in `--load` on the validation corpus. Run the\nabove command again to evaluate the perplexity of a pretrained model:\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\n## Step 4: Instruction tuning\n\nIn this step, we fine-tune the pretrained model on the downstream task with instructions. We provide a template\ninstruction tuning script to fine-tune 843M Retro.\n\nWe also provide an open-source blend of instruction tuning datasets. The dataset is available to download\nthrough [here](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing). The blendable\ndataset consists of the following open-source instruction tuning datasets:\n\n### Instruction Tuning Dataset Breakdown\n\n| Dataset                                                    | Samples | Epochs | Sampling Prob |\n|------------------------------------------------------------|--------:|-------:|--------------:|\n| [soda](https://arxiv.org/abs/2212.10465)                   |    2560 |  0.005 |         0.020 |\n| [eli5](https://arxiv.org/abs/1907.09190)                   |    2561 |  0.055 |         0.020 |\n| [self_instruct_short](https://arxiv.org/abs/2212.10560)    |    1280 |  0.043 |         0.010 |\n| [self_instruct_long](https://arxiv.org/abs/2212.10560)     |    2560 |  0.333 |         0.020 |\n| [unnatural-instructions](https://arxiv.org/abs/2212.09689) |    2560 |  0.024 |         0.020 |\n| [flan_cot](https://arxiv.org/abs/2210.11416)               |    1280 |  0.093 |         0.010 |\n| [dolly](https://arxiv.org/abs/2305.13735)                  |    6400 |  0.938 |         0.050 |\n| [oasst-skip-noncode](https://open-assistant.io/)           |  104558 |  1.839 |         0.817 |\n| [oasst-skip-code](https://open-assistant.io/)              |    4243 |  1.839 |         0.033 |\n\nRefer to the paper links above for more details about each instruction tuning dataset.\n\n*We note that the provided instruction tuning dataset is all from open-source instruction tuning datasets. It is\nslightly different from what we use in [InstructRetro](https://arxiv.org/abs/2310.07713), which contains private and\nproprietary datasets. Thus a 1-2% accuracy difference in downstream tasks may be expected.*\n\n### Instruction tuning script\n\nDownload\nthe [blended instruction tuning dataset](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing)\nin your data home directory `$DATA_HOME` and update our templates\nin [tools/retro/sft/sft_retro_lm.sh](sft/sft_retro_lm.sh).\n\nAn example command to run instruction tuning on 843M Retro is as follows:\n\n```bash\n                                      [blend-dataset-name] [model-size] [batch-size]  [lr]    [checkpoints]\nbash tools/retro/sft/sft_retro_lm.sh       open_inst               843m            128    5e-6  <path/to/pretrained/retro>\n```\n\nThe `blend_dataset_name` argument will blend all the datasets within the `$DATA_HOME` following the weights and\nconfigurations specified in the `${blend_dataset_name}.sh` ([open_inst.sh](sft/open_inst.sh) in the example above).\nThe checkpoints will be saved in the `--save` directory. For example, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6`.\n\n## Step 5: Downstream task evaluation\n\nIn this step, we demonstrate how to run InstructRetro for zero-shot evaluation on downstream question answering (QA)\ntasks. We provide the pre-processed open-source evaluation datasets with a unified format for different tasks. The\nevaluation datasets used in our paper are available to download\nthrough [here](https://drive.google.com/drive/folders/1xw-N0LJR_lIWnH6BKzHIb49quVCS_V72?usp=sharing). Please stick to\nthe same retro workdir used in Step 0-4 to make sure the preprocessed retrieval neighbors match the pretraining corpus.\nIf you directly come to Step 5, an example retro workdir with `args.json` for 800M Retro is\nprovided [here](https://drive.google.com/file/d/121GqAdMvf8bJEBZRt-SD4uhW-SRWgI3s/view?usp=sharing). Note that the args\nin the json can be overwritten through the command line.\n\nWe present an example command to run retro generation given the InstructRetro checkpoints and the Natural Question (NQ)\ntask. The example command is for the 843m InstructRetro obtained in Step 4. Please specify the directory for the NQ\ndataset and update the command accordingly for other checkpoints.\n\n```bash\nbash tools/retro/text_generation/retro_generate.sh nq 843m greedy test  0 20000 1000 5 pp1 <SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6 2\n```\n\nThe generated responses will be saved in the corresponding checkpoint directory. For example, for the 843m\nInstructRetro, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6/retro-generate-nq_5_2_843m_test_greedy_0_20000_1000.txt`.\n\nTo evaluate the F1 / Exact Match (EM) scores of the generated responses, we provide an example script to run the\nevaluation on the NQ dataset. Please specify the directory for the NQ dataset and update the command accordingly for\nother checkpoints and downstream tasks.\n\n```bash\npython3 tools/retro/text_generation/evaluate.py\n```\n\n# Citations\n\nSee more details from our papers:\n\n[Shall we Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study.](https://arxiv.org/abs/2304.06762)\n\n_Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei\nXiao, Anima Anandkumar, Bryan Catanzaro._ (EMNLP 2023)\n\n[InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining.](https://arxiv.org/abs/2310.07713)\n\n_Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, Bryan Catanzaro._\n\nPlease cite the papers as follows if you use the data or code from this repo:\n\n```bibtex\n@inproceedings{wang2023shall,\n    title   = {Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study},\n    author  = {Boxin Wang and Wei Ping and Peng Xu and Lawrence McAfee and Zihan Liu and Mohammad Shoeybi and Yi Dong and Oleksii Kuchaiev and Bo Li and Chaowei Xiao and Anima Anandkumar and Bryan Catanzaro},\n    journal = {The 2023 Conference on Empirical Methods in Natural Language Processing},\n    year    = {2023}\n}\n\n@article{wang2023instructretro,\n    title   = {InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining},\n    author  = {Boxin Wang and Wei Ping and Lawrence McAfee and Peng Xu and Bo Li and Mohammad Shoeybi and Bryan Catanzaro},\n    year    = {2023},\n    journal = {arXiv preprint arXiv: 2310.07713}\n}\n```\n","tools/retro/sft/README.md":"## Note\n\nThe content within this `sft` directory is still under active development and will be updated soon."},"files":{"tools/openwebtext/README.md":"The following steps show how to prepare training dataset to train the mode.\n\n# Libraries to install\n\n```\n    pip install ftfy langdetect numpy torch pandas nltk sentencepiece boto3 tqdm regex bs4 newspaper3k htmlmin tldextract \n    git clone https://github.com/mattilyra/LSH\n    cd LSH\n    python setup.py install\n``` \n\n# Download the dataset\n\n1. Download the deduplicated URLs from [jcpeterson](https://mega.nz/#F!EZZD0YwJ!9_PlEQzdMVLaNdKv_ICNVQ!cc4RgQQZ)\n2. Remove blacklisted URLs.\n```\npython blacklist_urls.py <path to the downloaded deduplicated URLs> <filename for clean urls. e.g. clean_urls.txt>\n```\n3. Download the content from the clean urls with [openwebtext's utilities](https://github.com/eukaryote31/openwebtext/blob/master/download.py). \n\n4. Merge the contents into one loose json file with 1 json per newline of the format `{'text': text, 'url': unique_url}`. It is important for the url to be unique.\n\n# Prepare the data for GPT training:\n\n1. Perform ftfy, english detection and remove documents with less than 128 tokens. This step can be sharded and run on shards.\n```\npython cleanup_dataset.py <input data file> <output cleaned data filename>\n```\nAdditional cleanup (e.g. remove documents less than 512 characters or dataset specific cleaning like stories, realnews datasets) can be done using `cleanup_fix_dataset.py`. More details can be found by running `python cleanup_fix_dataset.py --help`.\n2. Using LSH, find possible duplicates and store then in a file for later processing. The code supports saving and loading fingerprints for recurrent deduplications, and is also multithreaded for faster processing. More details are can be found by `python find_duplicate.py --help`.\n```\npython find_duplicates.py --inputs <pairlist list of input cleaned data files and keys, e.g. cc.json cc_id news.json news_id> --output <output possible duplicate urls filename>\n```\n3. Based on similarity measure defind inside function `is_similar` (default: 0.9), group urls that are similar. Basically, for each group, only one url we should keep and remove the rest.\n```\npython group_duplicate_urls.py <possible duplicate urls file> <output file containing similar urls>\n```\n4. Remove similar documents that were detected in the last step.\n```\npython remove_group_duplicates.py <file containing similar documents> <cleaned data file> <outputfile containing deduplicate data>\n```\n\n5. Shuffle the dataset.\n```\nshuf <cleaned deduped data file> -o train_data.json\n```\n\n# Deduplicating ngrams\n\nTo deduplicate the downstream tasks (e.g. lambada, squad) from the training dataset, we run the following command.\n\n```\npython filter_ngrams.py --tasks <name of the task, e.g. lambada, squad> --dedup-dataset <training dataset to deduplicate> <json key> --output <output training dataset>\n```\nWe use 13-grams by default for the deduplication. When we find a 13-gram match in a training document, we split the document into two pieces and remove the 13-gram along with 200 characters from the both side of the 13-gram. We also remove any splitted document with less than 200 characters or if a document got splitted more than 10 times. These parameters can be changed using corresponding arguments.\n\nOnly for the lambada task, we need to provide the path, `--lambada-path <path of the lambada test data>`.\n\nSeveral other features (e.g. save and load dictionary) have been added, look at `python filter_ngrams.py --help` for details.\n","tools/retro/README.md":"# Retro and InstructRetro\n\nRetro [(Borgeaud et al., 2022)](https://arxiv.org/abs/2112.04426) is an autoregressive decoder-only language model (LM)\npretrained with retrieval-augmentation.\nRetro features practical scalability to support large-scale pretraining from scratch by retrieving from trillions of\ntokens.\nPretraining with retrieval provides a more efficient storage mechanism of factual knowledge, when compared to storing\nfactual knowledge implicitly within the network's parameters, thus largely reducing model parameters while achieving\nlower perplexity than standard GPT.\nRetro also provides the flexibility to update the\nknowledge stored in LMs [(Wang et al., 2023a)](https://arxiv.org/abs/2304.06762)\nby updating the retrieval database without training LMs again.\n\nInstructRetro [(Wang et al., 2023b)](https://arxiv.org/abs/2310.07713) further scales up the size of Retro to 48B,\nfeaturing the largest LLM pretrained with retrieval (as of December 2023).\nThe obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity.\nWith instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on\ndownstream tasks in the zero-shot setting. Specifically, the average improvement of InstructRetro is 7% over its GPT\ncounterpart across 8 short-form QA tasks, 10% over GPT across 4 challenging long-form QA tasks, and 16% over GPT across\n3 summarization tasks. We also find that one can ablate the encoder from InstructRetro architecture and directly use the\nInstructRetro decoder backbone as GPT, while achieving comparable results.\n\nThis README provides an end-to-end tutorial to reproduce Retro and InstructRetro.\n\n# Contents\n\n* [Checkpoints](#checkpoints)\n* [End-to-end Reproduction Guide](#end-to-end-reproduction-guide)\n    * [Step 0: Prepare the environment](#step-0-prepare-the-environment)\n        * [Docker image](#docker-image)\n        * [Install dependencies](#install-dependencies)\n    * [Step 1: Build retrieval database](#step-1-build-retrieval-database)\n    * [Step 2: Pretraining](#step-2-pretraining)\n    * [Step 3: Perplexity evaluation](#step-3-perplexity-evaluation)\n    * [Step 4: Instruction tuning](#step-4-instruction-tuning)\n    * [Step 5: Downstream task evaluation](#step-5-downstream-task-evaluation)\n* [Citations](#citations)\n\n# Checkpoints\n\nWe provide the pretrained checkpoints of Retro and InstructRetro in the following table. The checkpoints are available\nto download through the following links:\n\n| Model                   | Size | Instruction Tuning | Download Link 1                                                    | Download Link 2                                                                | Download Link 3                                                                                      |\n|-------------------------|------|--------------------|--------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|\n| `retro-8b-base-4k`      | 8b   |                    | [Huggingface](https://huggingface.co/nvidia/retro-8b-base-4k)      | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-base-4k)      | [Google Drive](https://drive.google.com/drive/folders/1uSQ5DAsuvx_8XcbtnVfs_MGvEOcx0uK_?usp=sharing) |\n| `retro-8b-instruct-4k`  | 8b   | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-8b-instruct-4k)  | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-instruct-4k)  | [Google Drive](https://drive.google.com/drive/folders/1v5dKaSN0cm2lwyAWpFaJtlTrLhtMZXsI?usp=sharing) |\n| `retro-48b-base-4k`     | 48b  |                    | [Huggingface](https://huggingface.co/nvidia/retro-48b-base-4k)     | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-base-4k)     | [Google Drive](https://drive.google.com/drive/folders/1rtNpf0CiLElSHQcr3aLI3zgfI3teGTP5?usp=sharing) |\n| `retro-48b-instruct-4k` | 48b  | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-48b-instruct-4k) | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-instruct-4k) | [Google Drive](https://drive.google.com/drive/folders/1qdb0AQjSsAPGlWaIu3wgHPjf_nwLeY5h?usp=sharing) |\n\n# End-to-end Reproduction Guide\n\nIn this README, we provide an end-to-end reproduction guide for InstructRetro, covering from large-scale retrieval\nconstruction, pretraining, perplexity evaluation, instruction tuning, to downstream task evaluation.\n\nIf you are interested in evaluation only, we also [open-sourced our checkpoints](#checkpoints) and you can directly go\nto [Step 5](#step-5-downstream-task-evaluation) to evaluate the checkpoints on downstream tasks.\n\n## Step 0: Prepare the environment\n\nWe recommend using docker environment to run the code.\n\n### Docker image\n\nWe provide a docker build file in [tools/retro/examples/Dockerfile](examples/Dockerfile) for the reproduction. The\ndocker image is based on the [NGC docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) `nvcr.io/nvidia/pytorch:23.09-py3`.\n\n### Install dependencies\n\nClone the Megatron repo:\n\n```bash\ngit clone --branch InstructRetro https://github.com/NVIDIA/Megatron-LM.git\n```\n\nIf docker is not available, we recommend starting from a clean conda environment with the following runtime\ndependencies:\n\n- Python 3.10\n- NVIDIA CUDA® 12.2.1\n- NVIDIA cuBLAS 12.2.5.6\n- NVIDIA cuDNN 8.9.5\n- NVIDIA NCCL 2.18.5\n- PyTorch 2.1.0a0+32f93b1\n\nThen install Retro-specific dependencies, including:\n\n```bash\npip install -U faiss-gpu\npip install -U transformers\npip install -U sentencepiece\npip install -U h5py\npip install -U nltk\npip install -U einops\n```\n\n## Step 1: Build retrieval database\n\nIn this step, we build a large-scale retrieval database for InstructRetro\nthrough [Faiss](https://github.com/facebookresearch/faiss) to retrieve from trillions of tokens, and preprocess (and\nsave) the retrieval neighbors for the pretraining step.\n\nPlease refer to [tools/retro/build_db.md](build_db.md) for more details.\n\n## Step 2: Pretraining\n\n*Please strictly follow Step 1 to build the retrieval database before pretraining to make sure the preprocessed\nretrieval neighbors match the pretraining corpus.*\n\nIn the pretraining step, we support both pretraining from scratch and continued pretraining from a pretrained GPT model.\n\nWe provide a template pretraining script to pretrain 843M Retro from scratch. Prepare your own arguments and update our\ntemplates in [tools/retro/examples/pretrain_model.sh](examples/pretrain_model.sh). Please note that the data path should\nbe exactly matching the one used in Step 1 to make sure the preprocessed retrieval neighbors match the pretraining\ncorpus.\n\n[//]: # (Take the example of the Wikipedia corpus)\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\nAfter pretraining, the model checkpoints will be saved in the `--save` directory if you specified the arg\nin `pretrain_model.sh`.\n\nTo continue pretraining with retrieval from a pretrained GPT model, please specify `--load` in `pretrain_model.sh` to\nload the pretrained GPT model checkpoint (the architecture of GPT, including hidden size, number of layers, and\nactivation methods, should be exactly the same as the one used for Retro). You should also\nspecify  `--no-load-optim --finetune` to make sure the optimizer state is not loaded from the pretrained GPT model and\nthe continued pretraining with retrieval is from a clean start. After the first job / the first run, you will continue\npretraining with retrieval from your last checkpoint. In the follow-up jobs, you should launch the pretraining without\nthe flags `--no-load-optim --finetune` to make sure the optimizer state is correctly loaded from your last job.\n\n## Step 3: Perplexity evaluation\n\nDuring pretraining, we will automatically evaluate the model perplexity on the specified validation corpus\nevery `--eval-interval` steps. The validation corpus should be exactly the same as the one used in Step 1 to make sure\nthe preprocessed retrieval neighbors match the pretraining corpus.\n\nTo evaluate the perplexity of a pretrained model, please add `--skip-train` in `pretrain_model.sh` to skip the\npretraining step and only evaluate the perplexity of the model specified in `--load` on the validation corpus. Run the\nabove command again to evaluate the perplexity of a pretrained model:\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\n## Step 4: Instruction tuning\n\nIn this step, we fine-tune the pretrained model on the downstream task with instructions. We provide a template\ninstruction tuning script to fine-tune 843M Retro.\n\nWe also provide an open-source blend of instruction tuning datasets. The dataset is available to download\nthrough [here](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing). The blendable\ndataset consists of the following open-source instruction tuning datasets:\n\n### Instruction Tuning Dataset Breakdown\n\n| Dataset                                                    | Samples | Epochs | Sampling Prob |\n|------------------------------------------------------------|--------:|-------:|--------------:|\n| [soda](https://arxiv.org/abs/2212.10465)                   |    2560 |  0.005 |         0.020 |\n| [eli5](https://arxiv.org/abs/1907.09190)                   |    2561 |  0.055 |         0.020 |\n| [self_instruct_short](https://arxiv.org/abs/2212.10560)    |    1280 |  0.043 |         0.010 |\n| [self_instruct_long](https://arxiv.org/abs/2212.10560)     |    2560 |  0.333 |         0.020 |\n| [unnatural-instructions](https://arxiv.org/abs/2212.09689) |    2560 |  0.024 |         0.020 |\n| [flan_cot](https://arxiv.org/abs/2210.11416)               |    1280 |  0.093 |         0.010 |\n| [dolly](https://arxiv.org/abs/2305.13735)                  |    6400 |  0.938 |         0.050 |\n| [oasst-skip-noncode](https://open-assistant.io/)           |  104558 |  1.839 |         0.817 |\n| [oasst-skip-code](https://open-assistant.io/)              |    4243 |  1.839 |         0.033 |\n\nRefer to the paper links above for more details about each instruction tuning dataset.\n\n*We note that the provided instruction tuning dataset is all from open-source instruction tuning datasets. It is\nslightly different from what we use in [InstructRetro](https://arxiv.org/abs/2310.07713), which contains private and\nproprietary datasets. Thus a 1-2% accuracy difference in downstream tasks may be expected.*\n\n### Instruction tuning script\n\nDownload\nthe [blended instruction tuning dataset](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing)\nin your data home directory `$DATA_HOME` and update our templates\nin [tools/retro/sft/sft_retro_lm.sh](sft/sft_retro_lm.sh).\n\nAn example command to run instruction tuning on 843M Retro is as follows:\n\n```bash\n                                      [blend-dataset-name] [model-size] [batch-size]  [lr]    [checkpoints]\nbash tools/retro/sft/sft_retro_lm.sh       open_inst               843m            128    5e-6  <path/to/pretrained/retro>\n```\n\nThe `blend_dataset_name` argument will blend all the datasets within the `$DATA_HOME` following the weights and\nconfigurations specified in the `${blend_dataset_name}.sh` ([open_inst.sh](sft/open_inst.sh) in the example above).\nThe checkpoints will be saved in the `--save` directory. For example, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6`.\n\n## Step 5: Downstream task evaluation\n\nIn this step, we demonstrate how to run InstructRetro for zero-shot evaluation on downstream question answering (QA)\ntasks. We provide the pre-processed open-source evaluation datasets with a unified format for different tasks. The\nevaluation datasets used in our paper are available to download\nthrough [here](https://drive.google.com/drive/folders/1xw-N0LJR_lIWnH6BKzHIb49quVCS_V72?usp=sharing). Please stick to\nthe same retro workdir used in Step 0-4 to make sure the preprocessed retrieval neighbors match the pretraining corpus.\nIf you directly come to Step 5, an example retro workdir with `args.json` for 800M Retro is\nprovided [here](https://drive.google.com/file/d/121GqAdMvf8bJEBZRt-SD4uhW-SRWgI3s/view?usp=sharing). Note that the args\nin the json can be overwritten through the command line.\n\nWe present an example command to run retro generation given the InstructRetro checkpoints and the Natural Question (NQ)\ntask. The example command is for the 843m InstructRetro obtained in Step 4. Please specify the directory for the NQ\ndataset and update the command accordingly for other checkpoints.\n\n```bash\nbash tools/retro/text_generation/retro_generate.sh nq 843m greedy test  0 20000 1000 5 pp1 <SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6 2\n```\n\nThe generated responses will be saved in the corresponding checkpoint directory. For example, for the 843m\nInstructRetro, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6/retro-generate-nq_5_2_843m_test_greedy_0_20000_1000.txt`.\n\nTo evaluate the F1 / Exact Match (EM) scores of the generated responses, we provide an example script to run the\nevaluation on the NQ dataset. Please specify the directory for the NQ dataset and update the command accordingly for\nother checkpoints and downstream tasks.\n\n```bash\npython3 tools/retro/text_generation/evaluate.py\n```\n\n# Citations\n\nSee more details from our papers:\n\n[Shall we Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study.](https://arxiv.org/abs/2304.06762)\n\n_Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei\nXiao, Anima Anandkumar, Bryan Catanzaro._ (EMNLP 2023)\n\n[InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining.](https://arxiv.org/abs/2310.07713)\n\n_Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, Bryan Catanzaro._\n\nPlease cite the papers as follows if you use the data or code from this repo:\n\n```bibtex\n@inproceedings{wang2023shall,\n    title   = {Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study},\n    author  = {Boxin Wang and Wei Ping and Peng Xu and Lawrence McAfee and Zihan Liu and Mohammad Shoeybi and Yi Dong and Oleksii Kuchaiev and Bo Li and Chaowei Xiao and Anima Anandkumar and Bryan Catanzaro},\n    journal = {The 2023 Conference on Empirical Methods in Natural Language Processing},\n    year    = {2023}\n}\n\n@article{wang2023instructretro,\n    title   = {InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining},\n    author  = {Boxin Wang and Wei Ping and Lawrence McAfee and Peng Xu and Bo Li and Mohammad Shoeybi and Bryan Catanzaro},\n    year    = {2023},\n    journal = {arXiv preprint arXiv: 2310.07713}\n}\n```\n","tools/retro/sft/README.md":"## Note\n\nThe content within this `sft` directory is still under active development and will be updated soon."},"items":[{"name":"README.md","path":"tools/openwebtext/README.md","rawUrl":"https://raw.githubusercontent.com/NVIDIA/Megatron-LM/HEAD/tools/openwebtext/README.md","title":"openwebtext Documentation","category":"plugin-manifest","format":"markdown","content":"The following steps show how to prepare training dataset to train the mode.\n\n# Libraries to install\n\n```\n    pip install ftfy langdetect numpy torch pandas nltk sentencepiece boto3 tqdm regex bs4 newspaper3k htmlmin tldextract \n    git clone https://github.com/mattilyra/LSH\n    cd LSH\n    python setup.py install\n``` \n\n# Download the dataset\n\n1. Download the deduplicated URLs from [jcpeterson](https://mega.nz/#F!EZZD0YwJ!9_PlEQzdMVLaNdKv_ICNVQ!cc4RgQQZ)\n2. Remove blacklisted URLs.\n```\npython blacklist_urls.py <path to the downloaded deduplicated URLs> <filename for clean urls. e.g. clean_urls.txt>\n```\n3. Download the content from the clean urls with [openwebtext's utilities](https://github.com/eukaryote31/openwebtext/blob/master/download.py). \n\n4. Merge the contents into one loose json file with 1 json per newline of the format `{'text': text, 'url': unique_url}`. It is important for the url to be unique.\n\n# Prepare the data for GPT training:\n\n1. Perform ftfy, english detection and remove documents with less than 128 tokens. This step can be sharded and run on shards.\n```\npython cleanup_dataset.py <input data file> <output cleaned data filename>\n```\nAdditional cleanup (e.g. remove documents less than 512 characters or dataset specific cleaning like stories, realnews datasets) can be done using `cleanup_fix_dataset.py`. More details can be found by running `python cleanup_fix_dataset.py --help`.\n2. Using LSH, find possible duplicates and store then in a file for later processing. The code supports saving and loading fingerprints for recurrent deduplications, and is also multithreaded for faster processing. More details are can be found by `python find_duplicate.py --help`.\n```\npython find_duplicates.py --inputs <pairlist list of input cleaned data files and keys, e.g. cc.json cc_id news.json news_id> --output <output possible duplicate urls filename>\n```\n3. Based on similarity measure defind inside function `is_similar` (default: 0.9), group urls that are similar. Basically, for each group, only one url we should keep and remove the rest.\n```\npython group_duplicate_urls.py <possible duplicate urls file> <output file containing similar urls>\n```\n4. Remove similar documents that were detected in the last step.\n```\npython remove_group_duplicates.py <file containing similar documents> <cleaned data file> <outputfile containing deduplicate data>\n```\n\n5. Shuffle the dataset.\n```\nshuf <cleaned deduped data file> -o train_data.json\n```\n\n# Deduplicating ngrams\n\nTo deduplicate the downstream tasks (e.g. lambada, squad) from the training dataset, we run the following command.\n\n```\npython filter_ngrams.py --tasks <name of the task, e.g. lambada, squad> --dedup-dataset <training dataset to deduplicate> <json key> --output <output training dataset>\n```\nWe use 13-grams by default for the deduplication. When we find a 13-gram match in a training document, we split the document into two pieces and remove the 13-gram along with 200 characters from the both side of the 13-gram. We also remove any splitted document with less than 200 characters or if a document got splitted more than 10 times. These parameters can be changed using corresponding arguments.\n\nOnly for the lambada task, we need to provide the path, `--lambada-path <path of the lambada test data>`.\n\nSeveral other features (e.g. save and load dictionary) have been added, look at `python filter_ngrams.py --help` for details.\n","isInternal":false,"tokens":824,"sizeBytes":3426},{"name":"README.md","path":"tools/retro/README.md","rawUrl":"https://raw.githubusercontent.com/NVIDIA/Megatron-LM/HEAD/tools/retro/README.md","title":"retro Documentation","category":"plugin-manifest","format":"markdown","content":"# Retro and InstructRetro\n\nRetro [(Borgeaud et al., 2022)](https://arxiv.org/abs/2112.04426) is an autoregressive decoder-only language model (LM)\npretrained with retrieval-augmentation.\nRetro features practical scalability to support large-scale pretraining from scratch by retrieving from trillions of\ntokens.\nPretraining with retrieval provides a more efficient storage mechanism of factual knowledge, when compared to storing\nfactual knowledge implicitly within the network's parameters, thus largely reducing model parameters while achieving\nlower perplexity than standard GPT.\nRetro also provides the flexibility to update the\nknowledge stored in LMs [(Wang et al., 2023a)](https://arxiv.org/abs/2304.06762)\nby updating the retrieval database without training LMs again.\n\nInstructRetro [(Wang et al., 2023b)](https://arxiv.org/abs/2310.07713) further scales up the size of Retro to 48B,\nfeaturing the largest LLM pretrained with retrieval (as of December 2023).\nThe obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity.\nWith instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on\ndownstream tasks in the zero-shot setting. Specifically, the average improvement of InstructRetro is 7% over its GPT\ncounterpart across 8 short-form QA tasks, 10% over GPT across 4 challenging long-form QA tasks, and 16% over GPT across\n3 summarization tasks. We also find that one can ablate the encoder from InstructRetro architecture and directly use the\nInstructRetro decoder backbone as GPT, while achieving comparable results.\n\nThis README provides an end-to-end tutorial to reproduce Retro and InstructRetro.\n\n# Contents\n\n* [Checkpoints](#checkpoints)\n* [End-to-end Reproduction Guide](#end-to-end-reproduction-guide)\n    * [Step 0: Prepare the environment](#step-0-prepare-the-environment)\n        * [Docker image](#docker-image)\n        * [Install dependencies](#install-dependencies)\n    * [Step 1: Build retrieval database](#step-1-build-retrieval-database)\n    * [Step 2: Pretraining](#step-2-pretraining)\n    * [Step 3: Perplexity evaluation](#step-3-perplexity-evaluation)\n    * [Step 4: Instruction tuning](#step-4-instruction-tuning)\n    * [Step 5: Downstream task evaluation](#step-5-downstream-task-evaluation)\n* [Citations](#citations)\n\n# Checkpoints\n\nWe provide the pretrained checkpoints of Retro and InstructRetro in the following table. The checkpoints are available\nto download through the following links:\n\n| Model                   | Size | Instruction Tuning | Download Link 1                                                    | Download Link 2                                                                | Download Link 3                                                                                      |\n|-------------------------|------|--------------------|--------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|\n| `retro-8b-base-4k`      | 8b   |                    | [Huggingface](https://huggingface.co/nvidia/retro-8b-base-4k)      | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-base-4k)      | [Google Drive](https://drive.google.com/drive/folders/1uSQ5DAsuvx_8XcbtnVfs_MGvEOcx0uK_?usp=sharing) |\n| `retro-8b-instruct-4k`  | 8b   | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-8b-instruct-4k)  | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-instruct-4k)  | [Google Drive](https://drive.google.com/drive/folders/1v5dKaSN0cm2lwyAWpFaJtlTrLhtMZXsI?usp=sharing) |\n| `retro-48b-base-4k`     | 48b  |                    | [Huggingface](https://huggingface.co/nvidia/retro-48b-base-4k)     | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-base-4k)     | [Google Drive](https://drive.google.com/drive/folders/1rtNpf0CiLElSHQcr3aLI3zgfI3teGTP5?usp=sharing) |\n| `retro-48b-instruct-4k` | 48b  | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-48b-instruct-4k) | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-instruct-4k) | [Google Drive](https://drive.google.com/drive/folders/1qdb0AQjSsAPGlWaIu3wgHPjf_nwLeY5h?usp=sharing) |\n\n# End-to-end Reproduction Guide\n\nIn this README, we provide an end-to-end reproduction guide for InstructRetro, covering from large-scale retrieval\nconstruction, pretraining, perplexity evaluation, instruction tuning, to downstream task evaluation.\n\nIf you are interested in evaluation only, we also [open-sourced our checkpoints](#checkpoints) and you can directly go\nto [Step 5](#step-5-downstream-task-evaluation) to evaluate the checkpoints on downstream tasks.\n\n## Step 0: Prepare the environment\n\nWe recommend using docker environment to run the code.\n\n### Docker image\n\nWe provide a docker build file in [tools/retro/examples/Dockerfile](examples/Dockerfile) for the reproduction. The\ndocker image is based on the [NGC docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) `nvcr.io/nvidia/pytorch:23.09-py3`.\n\n### Install dependencies\n\nClone the Megatron repo:\n\n```bash\ngit clone --branch InstructRetro https://github.com/NVIDIA/Megatron-LM.git\n```\n\nIf docker is not available, we recommend starting from a clean conda environment with the following runtime\ndependencies:\n\n- Python 3.10\n- NVIDIA CUDA® 12.2.1\n- NVIDIA cuBLAS 12.2.5.6\n- NVIDIA cuDNN 8.9.5\n- NVIDIA NCCL 2.18.5\n- PyTorch 2.1.0a0+32f93b1\n\nThen install Retro-specific dependencies, including:\n\n```bash\npip install -U faiss-gpu\npip install -U transformers\npip install -U sentencepiece\npip install -U h5py\npip install -U nltk\npip install -U einops\n```\n\n## Step 1: Build retrieval database\n\nIn this step, we build a large-scale retrieval database for InstructRetro\nthrough [Faiss](https://github.com/facebookresearch/faiss) to retrieve from trillions of tokens, and preprocess (and\nsave) the retrieval neighbors for the pretraining step.\n\nPlease refer to [tools/retro/build_db.md](build_db.md) for more details.\n\n## Step 2: Pretraining\n\n*Please strictly follow Step 1 to build the retrieval database before pretraining to make sure the preprocessed\nretrieval neighbors match the pretraining corpus.*\n\nIn the pretraining step, we support both pretraining from scratch and continued pretraining from a pretrained GPT model.\n\nWe provide a template pretraining script to pretrain 843M Retro from scratch. Prepare your own arguments and update our\ntemplates in [tools/retro/examples/pretrain_model.sh](examples/pretrain_model.sh). Please note that the data path should\nbe exactly matching the one used in Step 1 to make sure the preprocessed retrieval neighbors match the pretraining\ncorpus.\n\n[//]: # (Take the example of the Wikipedia corpus)\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\nAfter pretraining, the model checkpoints will be saved in the `--save` directory if you specified the arg\nin `pretrain_model.sh`.\n\nTo continue pretraining with retrieval from a pretrained GPT model, please specify `--load` in `pretrain_model.sh` to\nload the pretrained GPT model checkpoint (the architecture of GPT, including hidden size, number of layers, and\nactivation methods, should be exactly the same as the one used for Retro). You should also\nspecify  `--no-load-optim --finetune` to make sure the optimizer state is not loaded from the pretrained GPT model and\nthe continued pretraining with retrieval is from a clean start. After the first job / the first run, you will continue\npretraining with retrieval from your last checkpoint. In the follow-up jobs, you should launch the pretraining without\nthe flags `--no-load-optim --finetune` to make sure the optimizer state is correctly loaded from your last job.\n\n## Step 3: Perplexity evaluation\n\nDuring pretraining, we will automatically evaluate the model perplexity on the specified validation corpus\nevery `--eval-interval` steps. The validation corpus should be exactly the same as the one used in Step 1 to make sure\nthe preprocessed retrieval neighbors match the pretraining corpus.\n\nTo evaluate the perplexity of a pretrained model, please add `--skip-train` in `pretrain_model.sh` to skip the\npretraining step and only evaluate the perplexity of the model specified in `--load` on the validation corpus. Run the\nabove command again to evaluate the perplexity of a pretrained model:\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\n## Step 4: Instruction tuning\n\nIn this step, we fine-tune the pretrained model on the downstream task with instructions. We provide a template\ninstruction tuning script to fine-tune 843M Retro.\n\nWe also provide an open-source blend of instruction tuning datasets. The dataset is available to download\nthrough [here](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing). The blendable\ndataset consists of the following open-source instruction tuning datasets:\n\n### Instruction Tuning Dataset Breakdown\n\n| Dataset                                                    | Samples | Epochs | Sampling Prob |\n|------------------------------------------------------------|--------:|-------:|--------------:|\n| [soda](https://arxiv.org/abs/2212.10465)                   |    2560 |  0.005 |         0.020 |\n| [eli5](https://arxiv.org/abs/1907.09190)                   |    2561 |  0.055 |         0.020 |\n| [self_instruct_short](https://arxiv.org/abs/2212.10560)    |    1280 |  0.043 |         0.010 |\n| [self_instruct_long](https://arxiv.org/abs/2212.10560)     |    2560 |  0.333 |         0.020 |\n| [unnatural-instructions](https://arxiv.org/abs/2212.09689) |    2560 |  0.024 |         0.020 |\n| [flan_cot](https://arxiv.org/abs/2210.11416)               |    1280 |  0.093 |         0.010 |\n| [dolly](https://arxiv.org/abs/2305.13735)                  |    6400 |  0.938 |         0.050 |\n| [oasst-skip-noncode](https://open-assistant.io/)           |  104558 |  1.839 |         0.817 |\n| [oasst-skip-code](https://open-assistant.io/)              |    4243 |  1.839 |         0.033 |\n\nRefer to the paper links above for more details about each instruction tuning dataset.\n\n*We note that the provided instruction tuning dataset is all from open-source instruction tuning datasets. It is\nslightly different from what we use in [InstructRetro](https://arxiv.org/abs/2310.07713), which contains private and\nproprietary datasets. Thus a 1-2% accuracy difference in downstream tasks may be expected.*\n\n### Instruction tuning script\n\nDownload\nthe [blended instruction tuning dataset](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing)\nin your data home directory `$DATA_HOME` and update our templates\nin [tools/retro/sft/sft_retro_lm.sh](sft/sft_retro_lm.sh).\n\nAn example command to run instruction tuning on 843M Retro is as follows:\n\n```bash\n                                      [blend-dataset-name] [model-size] [batch-size]  [lr]    [checkpoints]\nbash tools/retro/sft/sft_retro_lm.sh       open_inst               843m            128    5e-6  <path/to/pretrained/retro>\n```\n\nThe `blend_dataset_name` argument will blend all the datasets within the `$DATA_HOME` following the weights and\nconfigurations specified in the `${blend_dataset_name}.sh` ([open_inst.sh](sft/open_inst.sh) in the example above).\nThe checkpoints will be saved in the `--save` directory. For example, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6`.\n\n## Step 5: Downstream task evaluation\n\nIn this step, we demonstrate how to run InstructRetro for zero-shot evaluation on downstream question answering (QA)\ntasks. We provide the pre-processed open-source evaluation datasets with a unified format for different tasks. The\nevaluation datasets used in our paper are available to download\nthrough [here](https://drive.google.com/drive/folders/1xw-N0LJR_lIWnH6BKzHIb49quVCS_V72?usp=sharing). Please stick to\nthe same retro workdir used in Step 0-4 to make sure the preprocessed retrieval neighbors match the pretraining corpus.\nIf you directly come to Step 5, an example retro workdir with `args.json` for 800M Retro is\nprovided [here](https://drive.google.com/file/d/121GqAdMvf8bJEBZRt-SD4uhW-SRWgI3s/view?usp=sharing). Note that the args\nin the json can be overwritten through the command line.\n\nWe present an example command to run retro generation given the InstructRetro checkpoints and the Natural Question (NQ)\ntask. The example command is for the 843m InstructRetro obtained in Step 4. Please specify the directory for the NQ\ndataset and update the command accordingly for other checkpoints.\n\n```bash\nbash tools/retro/text_generation/retro_generate.sh nq 843m greedy test  0 20000 1000 5 pp1 <SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6 2\n```\n\nThe generated responses will be saved in the corresponding checkpoint directory. For example, for the 843m\nInstructRetro, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6/retro-generate-nq_5_2_843m_test_greedy_0_20000_1000.txt`.\n\nTo evaluate the F1 / Exact Match (EM) scores of the generated responses, we provide an example script to run the\nevaluation on the NQ dataset. Please specify the directory for the NQ dataset and update the command accordingly for\nother checkpoints and downstream tasks.\n\n```bash\npython3 tools/retro/text_generation/evaluate.py\n```\n\n# Citations\n\nSee more details from our papers:\n\n[Shall we Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study.](https://arxiv.org/abs/2304.06762)\n\n_Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei\nXiao, Anima Anandkumar, Bryan Catanzaro._ (EMNLP 2023)\n\n[InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining.](https://arxiv.org/abs/2310.07713)\n\n_Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, Bryan Catanzaro._\n\nPlease cite the papers as follows if you use the data or code from this repo:\n\n```bibtex\n@inproceedings{wang2023shall,\n    title   = {Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study},\n    author  = {Boxin Wang and Wei Ping and Peng Xu and Lawrence McAfee and Zihan Liu and Mohammad Shoeybi and Yi Dong and Oleksii Kuchaiev and Bo Li and Chaowei Xiao and Anima Anandkumar and Bryan Catanzaro},\n    journal = {The 2023 Conference on Empirical Methods in Natural Language Processing},\n    year    = {2023}\n}\n\n@article{wang2023instructretro,\n    title   = {InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining},\n    author  = {Boxin Wang and Wei Ping and Lawrence McAfee and Peng Xu and Bo Li and Mohammad Shoeybi and Bryan Catanzaro},\n    year    = {2023},\n    journal = {arXiv preprint arXiv: 2310.07713}\n}\n```\n","isInternal":false,"tokens":3938,"sizeBytes":14980},{"name":"README.md","path":"tools/retro/sft/README.md","rawUrl":"https://raw.githubusercontent.com/NVIDIA/Megatron-LM/HEAD/tools/retro/sft/README.md","title":"sft Documentation","category":"plugin-manifest","format":"markdown","content":"## Note\n\nThe content within this `sft` directory is still under active development and will be updated soon.","isInternal":false,"tokens":23,"sizeBytes":108}],"systemPromptSnippet":"<yakaai_skills repo=\"NVIDIA/Megatron-LM\">\n<!-- File: tools/openwebtext/README.md (Tokens: ~824 | Category: plugin-manifest) -->\nThe following steps show how to prepare training dataset to train the mode.\n\n# Libraries to install\n\n```\n    pip install ftfy langdetect numpy torch pandas nltk sentencepiece boto3 tqdm regex bs4 newspaper3k htmlmin tldextract \n    git clone https://github.com/mattilyra/LSH\n    cd LSH\n    python setup.py install\n``` \n\n# Download the dataset\n\n1. Download the deduplicated URLs from [jcpeterson](https://mega.nz/#F!EZZD0YwJ!9_PlEQzdMVLaNdKv_ICNVQ!cc4RgQQZ)\n2. Remove blacklisted URLs.\n```\npython blacklist_urls.py <path to the downloaded deduplicated URLs> <filename for clean urls. e.g. clean_urls.txt>\n```\n3. Download the content from the clean urls with [openwebtext's utilities](https://github.com/eukaryote31/openwebtext/blob/master/download.py). \n\n4. Merge the contents into one loose json file with 1 json per newline of the format `{'text': text, 'url': unique_url}`. It is important for the url to be unique.\n\n# Prepare the data for GPT training:\n\n1. Perform ftfy, english detection and remove documents with less than 128 tokens. This step can be sharded and run on shards.\n```\npython cleanup_dataset.py <input data file> <output cleaned data filename>\n```\nAdditional cleanup (e.g. remove documents less than 512 characters or dataset specific cleaning like stories, realnews datasets) can be done using `cleanup_fix_dataset.py`. More details can be found by running `python cleanup_fix_dataset.py --help`.\n2. Using LSH, find possible duplicates and store then in a file for later processing. The code supports saving and loading fingerprints for recurrent deduplications, and is also multithreaded for faster processing. More details are can be found by `python find_duplicate.py --help`.\n```\npython find_duplicates.py --inputs <pairlist list of input cleaned data files and keys, e.g. cc.json cc_id news.json news_id> --output <output possible duplicate urls filename>\n```\n3. Based on similarity measure defind inside function `is_similar` (default: 0.9), group urls that are similar. Basically, for each group, only one url we should keep and remove the rest.\n```\npython group_duplicate_urls.py <possible duplicate urls file> <output file containing similar urls>\n```\n4. Remove similar documents that were detected in the last step.\n```\npython remove_group_duplicates.py <file containing similar documents> <cleaned data file> <outputfile containing deduplicate data>\n```\n\n5. Shuffle the dataset.\n```\nshuf <cleaned deduped data file> -o train_data.json\n```\n\n# Deduplicating ngrams\n\nTo deduplicate the downstream tasks (e.g. lambada, squad) from the training dataset, we run the following command.\n\n```\npython filter_ngrams.py --tasks <name of the task, e.g. lambada, squad> --dedup-dataset <training dataset to deduplicate> <json key> --output <output training dataset>\n```\nWe use 13-grams by default for the deduplication. When we find a 13-gram match in a training document, we split the document into two pieces and remove the 13-gram along with 200 characters from the both side of the 13-gram. We also remove any splitted document with less than 200 characters or if a document got splitted more than 10 times. These parameters can be changed using corresponding arguments.\n\nOnly for the lambada task, we need to provide the path, `--lambada-path <path of the lambada test data>`.\n\nSeveral other features (e.g. save and load dictionary) have been added, look at `python filter_ngrams.py --help` for details.\n\n<!-- File: tools/retro/README.md (Tokens: ~3938 | Category: plugin-manifest) -->\n# Retro and InstructRetro\n\nRetro [(Borgeaud et al., 2022)](https://arxiv.org/abs/2112.04426) is an autoregressive decoder-only language model (LM)\npretrained with retrieval-augmentation.\nRetro features practical scalability to support large-scale pretraining from scratch by retrieving from trillions of\ntokens.\nPretraining with retrieval provides a more efficient storage mechanism of factual knowledge, when compared to storing\nfactual knowledge implicitly within the network's parameters, thus largely reducing model parameters while achieving\nlower perplexity than standard GPT.\nRetro also provides the flexibility to update the\nknowledge stored in LMs [(Wang et al., 2023a)](https://arxiv.org/abs/2304.06762)\nby updating the retrieval database without training LMs again.\n\nInstructRetro [(Wang et al., 2023b)](https://arxiv.org/abs/2310.07713) further scales up the size of Retro to 48B,\nfeaturing the largest LLM pretrained with retrieval (as of December 2023).\nThe obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity.\nWith instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on\ndownstream tasks in the zero-shot setting. Specifically, the average improvement of InstructRetro is 7% over its GPT\ncounterpart across 8 short-form QA tasks, 10% over GPT across 4 challenging long-form QA tasks, and 16% over GPT across\n3 summarization tasks. We also find that one can ablate the encoder from InstructRetro architecture and directly use the\nInstructRetro decoder backbone as GPT, while achieving comparable results.\n\nThis README provides an end-to-end tutorial to reproduce Retro and InstructRetro.\n\n# Contents\n\n* [Checkpoints](#checkpoints)\n* [End-to-end Reproduction Guide](#end-to-end-reproduction-guide)\n    * [Step 0: Prepare the environment](#step-0-prepare-the-environment)\n        * [Docker image](#docker-image)\n        * [Install dependencies](#install-dependencies)\n    * [Step 1: Build retrieval database](#step-1-build-retrieval-database)\n    * [Step 2: Pretraining](#step-2-pretraining)\n    * [Step 3: Perplexity evaluation](#step-3-perplexity-evaluation)\n    * [Step 4: Instruction tuning](#step-4-instruction-tuning)\n    * [Step 5: Downstream task evaluation](#step-5-downstream-task-evaluation)\n* [Citations](#citations)\n\n# Checkpoints\n\nWe provide the pretrained checkpoints of Retro and InstructRetro in the following table. The checkpoints are available\nto download through the following links:\n\n| Model                   | Size | Instruction Tuning | Download Link 1                                                    | Download Link 2                                                                | Download Link 3                                                                                      |\n|-------------------------|------|--------------------|--------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|\n| `retro-8b-base-4k`      | 8b   |                    | [Huggingface](https://huggingface.co/nvidia/retro-8b-base-4k)      | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-base-4k)      | [Google Drive](https://drive.google.com/drive/folders/1uSQ5DAsuvx_8XcbtnVfs_MGvEOcx0uK_?usp=sharing) |\n| `retro-8b-instruct-4k`  | 8b   | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-8b-instruct-4k)  | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-8b-instruct-4k)  | [Google Drive](https://drive.google.com/drive/folders/1v5dKaSN0cm2lwyAWpFaJtlTrLhtMZXsI?usp=sharing) |\n| `retro-48b-base-4k`     | 48b  |                    | [Huggingface](https://huggingface.co/nvidia/retro-48b-base-4k)     | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-base-4k)     | [Google Drive](https://drive.google.com/drive/folders/1rtNpf0CiLElSHQcr3aLI3zgfI3teGTP5?usp=sharing) |\n| `retro-48b-instruct-4k` | 48b  | ✅                  | [Huggingface](https://huggingface.co/nvidia/retro-48b-instruct-4k) | [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/models/retro-48b-instruct-4k) | [Google Drive](https://drive.google.com/drive/folders/1qdb0AQjSsAPGlWaIu3wgHPjf_nwLeY5h?usp=sharing) |\n\n# End-to-end Reproduction Guide\n\nIn this README, we provide an end-to-end reproduction guide for InstructRetro, covering from large-scale retrieval\nconstruction, pretraining, perplexity evaluation, instruction tuning, to downstream task evaluation.\n\nIf you are interested in evaluation only, we also [open-sourced our checkpoints](#checkpoints) and you can directly go\nto [Step 5](#step-5-downstream-task-evaluation) to evaluate the checkpoints on downstream tasks.\n\n## Step 0: Prepare the environment\n\nWe recommend using docker environment to run the code.\n\n### Docker image\n\nWe provide a docker build file in [tools/retro/examples/Dockerfile](examples/Dockerfile) for the reproduction. The\ndocker image is based on the [NGC docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) `nvcr.io/nvidia/pytorch:23.09-py3`.\n\n### Install dependencies\n\nClone the Megatron repo:\n\n```bash\ngit clone --branch InstructRetro https://github.com/NVIDIA/Megatron-LM.git\n```\n\nIf docker is not available, we recommend starting from a clean conda environment with the following runtime\ndependencies:\n\n- Python 3.10\n- NVIDIA CUDA® 12.2.1\n- NVIDIA cuBLAS 12.2.5.6\n- NVIDIA cuDNN 8.9.5\n- NVIDIA NCCL 2.18.5\n- PyTorch 2.1.0a0+32f93b1\n\nThen install Retro-specific dependencies, including:\n\n```bash\npip install -U faiss-gpu\npip install -U transformers\npip install -U sentencepiece\npip install -U h5py\npip install -U nltk\npip install -U einops\n```\n\n## Step 1: Build retrieval database\n\nIn this step, we build a large-scale retrieval database for InstructRetro\nthrough [Faiss](https://github.com/facebookresearch/faiss) to retrieve from trillions of tokens, and preprocess (and\nsave) the retrieval neighbors for the pretraining step.\n\nPlease refer to [tools/retro/build_db.md](build_db.md) for more details.\n\n## Step 2: Pretraining\n\n*Please strictly follow Step 1 to build the retrieval database before pretraining to make sure the preprocessed\nretrieval neighbors match the pretraining corpus.*\n\nIn the pretraining step, we support both pretraining from scratch and continued pretraining from a pretrained GPT model.\n\nWe provide a template pretraining script to pretrain 843M Retro from scratch. Prepare your own arguments and update our\ntemplates in [tools/retro/examples/pretrain_model.sh](examples/pretrain_model.sh). Please note that the data path should\nbe exactly matching the one used in Step 1 to make sure the preprocessed retrieval neighbors match the pretraining\ncorpus.\n\n[//]: # (Take the example of the Wikipedia corpus)\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\nAfter pretraining, the model checkpoints will be saved in the `--save` directory if you specified the arg\nin `pretrain_model.sh`.\n\nTo continue pretraining with retrieval from a pretrained GPT model, please specify `--load` in `pretrain_model.sh` to\nload the pretrained GPT model checkpoint (the architecture of GPT, including hidden size, number of layers, and\nactivation methods, should be exactly the same as the one used for Retro). You should also\nspecify  `--no-load-optim --finetune` to make sure the optimizer state is not loaded from the pretrained GPT model and\nthe continued pretraining with retrieval is from a clean start. After the first job / the first run, you will continue\npretraining with retrieval from your last checkpoint. In the follow-up jobs, you should launch the pretraining without\nthe flags `--no-load-optim --finetune` to make sure the optimizer state is correctly loaded from your last job.\n\n## Step 3: Perplexity evaluation\n\nDuring pretraining, we will automatically evaluate the model perplexity on the specified validation corpus\nevery `--eval-interval` steps. The validation corpus should be exactly the same as the one used in Step 1 to make sure\nthe preprocessed retrieval neighbors match the pretraining corpus.\n\nTo evaluate the perplexity of a pretrained model, please add `--skip-train` in `pretrain_model.sh` to skip the\npretraining step and only evaluate the perplexity of the model specified in `--load` on the validation corpus. Run the\nabove command again to evaluate the perplexity of a pretrained model:\n\n```bash\nbash tools/retro/examples/pretrain_model.sh\n```\n\n## Step 4: Instruction tuning\n\nIn this step, we fine-tune the pretrained model on the downstream task with instructions. We provide a template\ninstruction tuning script to fine-tune 843M Retro.\n\nWe also provide an open-source blend of instruction tuning datasets. The dataset is available to download\nthrough [here](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing). The blendable\ndataset consists of the following open-source instruction tuning datasets:\n\n### Instruction Tuning Dataset Breakdown\n\n| Dataset                                                    | Samples | Epochs | Sampling Prob |\n|------------------------------------------------------------|--------:|-------:|--------------:|\n| [soda](https://arxiv.org/abs/2212.10465)                   |    2560 |  0.005 |         0.020 |\n| [eli5](https://arxiv.org/abs/1907.09190)                   |    2561 |  0.055 |         0.020 |\n| [self_instruct_short](https://arxiv.org/abs/2212.10560)    |    1280 |  0.043 |         0.010 |\n| [self_instruct_long](https://arxiv.org/abs/2212.10560)     |    2560 |  0.333 |         0.020 |\n| [unnatural-instructions](https://arxiv.org/abs/2212.09689) |    2560 |  0.024 |         0.020 |\n| [flan_cot](https://arxiv.org/abs/2210.11416)               |    1280 |  0.093 |         0.010 |\n| [dolly](https://arxiv.org/abs/2305.13735)                  |    6400 |  0.938 |         0.050 |\n| [oasst-skip-noncode](https://open-assistant.io/)           |  104558 |  1.839 |         0.817 |\n| [oasst-skip-code](https://open-assistant.io/)              |    4243 |  1.839 |         0.033 |\n\nRefer to the paper links above for more details about each instruction tuning dataset.\n\n*We note that the provided instruction tuning dataset is all from open-source instruction tuning datasets. It is\nslightly different from what we use in [InstructRetro](https://arxiv.org/abs/2310.07713), which contains private and\nproprietary datasets. Thus a 1-2% accuracy difference in downstream tasks may be expected.*\n\n### Instruction tuning script\n\nDownload\nthe [blended instruction tuning dataset](https://drive.google.com/file/d/1nzKwwYf8lYb9gN3P4YO8pFNU_B2nMYe1/view?usp=sharing)\nin your data home directory `$DATA_HOME` and update our templates\nin [tools/retro/sft/sft_retro_lm.sh](sft/sft_retro_lm.sh).\n\nAn example command to run instruction tuning on 843M Retro is as follows:\n\n```bash\n                                      [blend-dataset-name] [model-size] [batch-size]  [lr]    [checkpoints]\nbash tools/retro/sft/sft_retro_lm.sh       open_inst               843m            128    5e-6  <path/to/pretrained/retro>\n```\n\nThe `blend_dataset_name` argument will blend all the datasets within the `$DATA_HOME` following the weights and\nconfigurations specified in the `${blend_dataset_name}.sh` ([open_inst.sh](sft/open_inst.sh) in the example above).\nThe checkpoints will be saved in the `--save` directory. For example, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6`.\n\n## Step 5: Downstream task evaluation\n\nIn this step, we demonstrate how to run InstructRetro for zero-shot evaluation on downstream question answering (QA)\ntasks. We provide the pre-processed open-source evaluation datasets with a unified format for different tasks. The\nevaluation datasets used in our paper are available to download\nthrough [here](https://drive.google.com/drive/folders/1xw-N0LJR_lIWnH6BKzHIb49quVCS_V72?usp=sharing). Please stick to\nthe same retro workdir used in Step 0-4 to make sure the preprocessed retrieval neighbors match the pretraining corpus.\nIf you directly come to Step 5, an example retro workdir with `args.json` for 800M Retro is\nprovided [here](https://drive.google.com/file/d/121GqAdMvf8bJEBZRt-SD4uhW-SRWgI3s/view?usp=sharing). Note that the args\nin the json can be overwritten through the command line.\n\nWe present an example command to run retro generation given the InstructRetro checkpoints and the Natural Question (NQ)\ntask. The example command is for the 843m InstructRetro obtained in Step 4. Please specify the directory for the NQ\ndataset and update the command accordingly for other checkpoints.\n\n```bash\nbash tools/retro/text_generation/retro_generate.sh nq 843m greedy test  0 20000 1000 5 pp1 <SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6 2\n```\n\nThe generated responses will be saved in the corresponding checkpoint directory. For example, for the 843m\nInstructRetro, it will be saved to\n`<SFT_HOME>/checkpoints/applications/retro-sft_pp1_same_format_ctx1_843m_128_5e-6/retro-generate-nq_5_2_843m_test_greedy_0_20000_1000.txt`.\n\nTo evaluate the F1 / Exact Match (EM) scores of the generated responses, we provide an example script to run the\nevaluation on the NQ dataset. Please specify the directory for the NQ dataset and update the command accordingly for\nother checkpoints and downstream tasks.\n\n```bash\npython3 tools/retro/text_generation/evaluate.py\n```\n\n# Citations\n\nSee more details from our papers:\n\n[Shall we Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study.](https://arxiv.org/abs/2304.06762)\n\n_Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei\nXiao, Anima Anandkumar, Bryan Catanzaro._ (EMNLP 2023)\n\n[InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining.](https://arxiv.org/abs/2310.07713)\n\n_Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, Bryan Catanzaro._\n\nPlease cite the papers as follows if you use the data or code from this repo:\n\n```bibtex\n@inproceedings{wang2023shall,\n    title   = {Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study},\n    author  = {Boxin Wang and Wei Ping and Peng Xu and Lawrence McAfee and Zihan Liu and Mohammad Shoeybi and Yi Dong and Oleksii Kuchaiev and Bo Li and Chaowei Xiao and Anima Anandkumar and Bryan Catanzaro},\n    journal = {The 2023 Conference on Empirical Methods in Natural Language Processing},\n    year    = {2023}\n}\n\n@article{wang2023instructretro,\n    title   = {InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining},\n    author  = {Boxin Wang and Wei Ping and Lawrence McAfee and Peng Xu and Bo Li and Mohammad Shoeybi and Bryan Catanzaro},\n    year    = {2023},\n    journal = {arXiv preprint arXiv: 2310.07713}\n}\n```\n\n<!-- File: tools/retro/sft/README.md (Tokens: ~23 | Category: plugin-manifest) -->\n## Note\n\nThe content within this `sft` directory is still under active development and will be updated soon.\n</yakaai_skills>"}