Huatuo-Llama-Med-Chinese Documentation

Repo for BenCao [original name: HuaTuo (华驼)], Instruction-tuning Large Language Models with Chinese Medical Knowledge. 本草（原名：华驼）模型仓库，基于中文医学知识的大语言模型指令微调

5,001 stars Python Python 3 files · ~6,812 tokens #aidoctor#bloom#chinese#huozi#llama#llm#medgpt#medical
RAW Doc

File: README.md

中文 | English

本草[原名：华驼(HuaTuo)]: 基于中文医学知识的大语言模型指令微调

BenTsao (original name: HuaTuo): Instruction-tuning Large Language Models With Chinese Medical Knowledge

本项目开源了经过中文医学指令精调/指令微调(Instruction-tuning) 的大语言模型集，包括LLaMA、Alpaca-Chinese、Bloom、活字模型等。

我们基于医学知识图谱以及医学文献，结合ChatGPT API构建了中文医学指令微调数据集，并以此对各种基模型进行了指令微调，提高了基模型在医疗领域的问答效果。

News

[2024/07/25]《基于知识微调的大语言模型可靠中文医学回复生成方法》被ACM TKDD录用

[2023/09/24]发布《面向智慧医疗的大语言模型微调技术》

历史新闻

[2023/09/12]在arxiv发布《探索大模型从医学文献中交互式知识的获取》

[2023/09/08]在arxiv发布《基于知识微调的大语言模型可靠中文医学回复生成方法》

[2023/08/07] 🔥🔥增加了基于活字进行指令微调的模型发布，模型效果显著提升。🔥🔥

[2023/08/05] 本草模型在CCL 2023 Demo Track进行Poster展示。

[2023/08/03] SCIR实验室开源活字通用问答模型，欢迎大家关注🎉🎉

[2023/07/19] 增加了基于Bloom进行指令微调的模型发布。

[2023/05/12] 模型由"华驼"更名为"本草"。

[2023/04/28] 增加了基于中文Alpaca大模型进行指令微调的模型发布。

[2023/04/24] 增加了基于LLaMA和医学文献进行指令微调的模型发布。

[2023/03/31] 增加了基于LLaMA和医学知识库进行指令微调的模型发布。

A Quick Start

首先安装依赖包，python环境建议3.9+

text
pip install -r requirements.txt

针对所有基模型，我们采用了半精度基模型LoRA微调的方式进行指令微调训练，以在计算资源与模型性能之间进行权衡。

基模型

LoRA模型权重下载

LoRA权重可以通过百度网盘或Hugging Face下载：

  1. 🔥对活字进行指令微调的LoRA权重文件
  • 基于医学知识库以及医学问答数据集 百度网盘
  1. 对Bloom进行指令微调的LoRA权重文件
  1. 对Alpaca进行指令微调的LoRA权重文件
  1. 对LLaMA进行指令微调的LoRA权重文件

下载LoRA权重并解压，解压后的格式如下：

text
**lora-folder-name**/
  - adapter_config.json   # LoRA权重配置文件
  - adapter_model.bin   # LoRA权重文件

基于相同的数据，我们还训练了医疗版本的ChatGLM模型: ChatGLM-6B-Med

Infer

我们在./data/infer.json中提供了一些测试用例，可以替换成其它的数据集，请注意保持格式一致

运行infer脚本

text
#基于医学知识库
bash ./scripts/infer.sh

#基于医学文献
#单轮
bash ./scripts/infer-literature-single.sh

#多轮
bash ./scripts/infer-literature-multi.sh

infer.sh脚本代码如下，请将下列代码中基模型base_model、lora权重lora_weights以及测试数据集路径instruct_dir进行替换后运行

text
python infer.py \
	    --base_model 'BASE_MODEL_PATH' \
	    --lora_weights 'LORA_WEIGHTS_PATH' \
	    --use_lora True \
	    --instruct_dir 'INFER_DATA_PATH' \
	    --prompt_template 'TEMPLATE_PATH'

提示模板的选择与模型相关，详情如下：

活字&Bloom LLaMA&Alpaca
templates/bloom_deploy.json 基于医学知识库templates/med_template.json 基于医学文献templates/literature_template.json

也可参考./scripts/test.sh

方法

基模型在医学问答场景下效果有限，指令微调是一种高效的使基模型拥有回答人类问题能力的方法。

数据集构建

医学知识库

我们采用了公开和自建的中文医学知识库，主要参考了cMeKG。

医学知识库围绕疾病、药物、检查指标等构建，字段包括并发症，高危因素，组织学检查，临床症状，药物治疗，辅助治疗等。知识库示例如下:

text
{"中心词": "偏头痛", "相关疾病": ["妊娠合并偏头痛", "恶寒发热"], "相关症状": ["皮肤变硬", "头部及眼后部疼痛并能听到连续不断的隆隆声", "晨起头痛加重"], "所属科室": ["中西医结合科", "内科"], "发病部位": ["头部"]}

我们利用GPT3.5接口围绕医学知识库构建问答数据，设置了多种Prompt形式来充分利用知识。

指令微调的训练集数据示例如下：

text
"问题：一位年轻男性长期使用可卡因，突然出现胸痛、呕吐、出汗等症状，经检查发现心电图反映心肌急性损伤，请问可能患的是什么疾病？治疗方式是什么？"

回答: 可能患的是心肌梗塞，需要进行维拉帕米、依普利酮、硝酸甘油、ß阻滞剂、吗啡等药物治疗，并进行溶栓治疗、低分子量肝素、钙通道阻滞剂等辅助治疗。此外需要及时停用可卡因等药物，以防止病情加重。"

我们提供了模型的训练数据集，共计八千余条，需要注意的是，虽然训练集的构建融入了知识，但是仍存在错误和不完善的地方，后续我们会利用更好的策略迭代更新数据集。

指令微调数据集质量仍有限，后续将进行不断迭代，同时医学知识库和数据集构建代码还在整理中，整理完成将会发布。

医学文献

此外，我们收集了2023年关于肝癌疾病的中文医学文献，利用GPT3.5接口围绕医学文献的【结论】构建多轮问答数据。在·./data_literature/liver_cancer.json中我们提供了其中的1k条训练样例。目前，训练样本的质量仍然有限，在后续我们会进一步迭代数据，会以公开数据集的形式对外进行发布。训练样本的示例如下：

目前，我们只开放针对"肝癌"单个疾病训练的模型参数。在未来，我们计划发布融入文献结论的医学对话数据集，并且会针对“肝胆胰”相关16种疾病训练模型。

相关细节可参考我们的文章：《探索大模型从医学文献中交互式知识的获取》

Finetune

如果想用自己的数据集微调大语言模型，请按照./data/llama_data.json中的格式构建自己的数据集

运行finetune脚本

text
bash ./scripts/finetune.sh

知识微调

上述介绍了我们如何利用医学知识构造高质量问答对的过程，即从“knowledge”到“(question, answer)”，在此基础上，我们提出了知识微调，旨在让大语言模型在推理时显式利用知识库中的知识，使大模型学会从“question”到“knowledge”，过程可分为以下三阶段，其过程如下图所示。第一阶段：根据问题填充知识检索的参数，包括中心词和属性。第二阶段：查询获取带有填充参数所对应的知识。第三阶段：使用获得的知识来生成回答。样例数据链接

相关细节可参考我们的文章：《基于知识微调的大语言模型可靠中文医学回复生成方法》

训练细节

计算资源需求参考

基于LLaMA模型的指令微调过程中，我们在一张A100-SXM-80GB显卡上进行了训练，训练总轮次10轮，耗时约2h17m。batch_size=128的情况下显存占用在40G左右。预计3090/4090显卡(24GB显存)以上显卡可以较好支持，根据显存大小来调整batch_size。

实验过程

超参设置和实验过程可以参考wandb的链接:

https://wandb.ai/thinksoso/llama_med/runs/a5wgcnzt/overview?workspace=user-thinksoso

模型效果对比

注：测试于2023年3月

测试输入 Llama输出 Alpaca输出 BenTsao本草输出
小张最近感觉身体不适，出现心悸、气促等症状。体检发现心脏扩大、搏动减弱。 问题：小张最近感觉身体不适，出现心悸、气促等症状。体检发现心脏扩大、搏动减弱。 最近，小张一直感觉身体不舒服，有心烦意乱、呼吸急促等症状。体格检查显示心脏增大，肌肉无力。 小张可能患有心肌炎，建议尽快到医院进行检查和治疗。治疗方案包括药物治疗和手术治疗。
麻风病和儿童哮喘的病因是否一致？ 儿童哮喘和麻风病的病因是一致，但是儿童哮喘和麻风病的病因不一致 是的，哮喘和儿童喘息有相似的原因。 不一致，麻风病的病因是麻风杆菌感染，而儿童哮喘的病因是喂养不良、吸烟等因素。
一位患者出现了持续性的胀痛和黄疸，经影像学检查后确诊为肝胆管结石病，应该采取哪些治疗方案？ 肝胆管结石病患者应该采取哪些治疗方案？ 患者应接受手术和化疗相结合的治疗。 针对肝胆管结石病，治疗方案包括手术治疗、药物治疗和辅助治疗等。具体治疗方案需要根据患者的具体情况而定。

常见问题

  1. Q: 为什么更名为"本草"？

    A: 为SCIR实验室大语言模型命名一致性，中文医学大模型名称调整为"本草"。

  2. Q: 为什么是"华驼"而不是"华佗"？

    A: 叫”驼“是因为我们的基模型LLaMA是美洲驼，Alpaca是羊驼，受他们名字的启发以及华佗的谐音梗，我们将我们的模型起名为华驼。

  3. Q: 有使用中医理论或者中医数据吗？

    A: 目前还没有

  4. Q: 模型运行的结果不同、效果有限

    A: 由于生成模型生成多样性的考量，多次运行的结果可能会有差异。当前开源的模型由于LLaMA及Alpaca中文语料有限，且知识结合的方式较为粗糙，请大家尝试bloom-based和活字-based的模型。

  5. Q: 模型无法运行/推理内容完全无法接受

    A: 请确定已安装requirements中的依赖、配置好cuda环境并添加环境变量、正确输入下载好的模型以及lora的存储位置；推理内容如存在重复生成或部分错误内容属于llama-based模型的偶发现象，与llama模型的中文能力、训练数据规模以及超参设置均有一定的关系，请尝试基于活字的新模型。如存在严重问题，请将运行的文件名、模型名、lora等配置信息详细描述在issue中，谢谢大家。

  6. Q: 发布的若干模型哪个最好？

    A: 根据我们的经验，基于活字模型的效果相对更好一些。

项目参与者

本项目由哈尔滨工业大学社会计算与信息检索研究中心健康智能组王昊淳 、杜晏睿、刘驰、白睿、席奴瓦、陈雨晗、强泽文、陈健宇、李子健、范宇政、刘佳彤完成，指导教师为赵森栋副教授，秦兵教授以及刘挺教授。

致谢

本项目参考了以下开源项目，在此对相关项目和研究开发人员表示感谢。

免责声明

本项目相关资源仅供学术研究之用，严禁用于商业用途。使用涉及第三方代码的部分时，请严格遵循相应的开源协议。模型生成的内容受模型计算、随机性和量化精度损失等因素影响，本项目无法对其准确性作出保证。本项目数据集绝大部分由模型生成，即使符合某些医学事实，也不能被用作实际医学诊断的依据。对于模型输出的任何内容，本项目不承担任何法律责任，亦不对因使用相关资源和输出结果而可能产生的任何损失承担责任。

Citation

如果您使用了本项目的数据或者代码，或是我们的工作对您有所帮助，请声明引用

知识微调：Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Trustworthy Response Generation in Chinese

text
@article{10.1145/3686807,
author = {Wang, Haochun and Zhao, Sendong and Qiang, Zewen and Li, Zijian and Liu, Chi and Xi, Nuwa and Du, Yanrui and Qin, Bing and Liu, Ting},
title = {Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Trustworthy Response Generation in Chinese},
year = {2024},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
issn = {1556-4681},
url = {https://doi.org/10.1145/3686807},
doi = {10.1145/3686807},
abstract = {Large Language Models (LLMs) have demonstrated remarkable success in diverse natural language processing (NLP) tasks in general domains. However, LLMs sometimes generate responses with the hallucination about medical facts due to limited domain knowledge. Such shortcomings pose potential risks in the utilization of LLMs within medical contexts. To address this challenge, we propose knowledge-tuning, which leverages structured medical knowledge bases for the LLMs to grasp domain knowledge efficiently and facilitate trustworthy response generation. We also release cMedKnowQA, a Chinese medical knowledge question-answering dataset constructed from medical knowledge bases to assess the medical knowledge proficiency of LLMs. Experimental results show that the LLMs which are knowledge-tuned with cMedKnowQA, can exhibit higher levels of accuracy in response generation compared with vanilla instruction-tuning and offer a new trustworthy way for the domain adaptation of LLMs. We release our code and data at .},
note = {Just Accepted},
journal = {ACM Trans. Knowl. Discov. Data},
month = aug,
keywords = {Large Language Model, Medical Question Answering, Trustworthy Response Generation, Medical Knowledge Base}
}

首版技术报告: Huatuo: Tuning llama model with chinese medical knowledge

text
@misc{wang2023huatuo,
      title={HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge},
      author={Haochun Wang and Chi Liu and Nuwa Xi and Zewen Qiang and Sendong Zhao and Bing Qin and Ting Liu},
      year={2023},
      eprint={2304.06975},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

医学文献知识获取：The CALLA Dataset: Probing LLMs’ Interactive Knowledge Acquisition from Chinese Medical Literature

text
@misc{du2023calla,
      title={The CALLA Dataset: Probing LLMs' Interactive Knowledge Acquisition from Chinese Medical Literature}, 
      author={Yanrui Du and Sendong Zhao and Muzhen Cai and Jianyu Chen and Haochun Wang and Yuhan Chen and Haoqiang Guo and Bing Qin},
      year={2023},
      eprint={2309.04198},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

File: README_EN.md

中文 | English

BenTsao (original name: HuaTuo): Instruction-tuning Large Language Models With Chinese Medical Knowledge

This repo open-sources a series of instruction-tuned large language models with Chinese medical instruction datasets, including LLaMA、Alpaca-Chinese、Bloom、Huozi.

We constructed a Chinese medical instruct-tuning dataset using medical knowledge graphs, medical literatures and the GPT3.5 API, and performed instruction-tuning on various base models with this dataset, improving its question-answering performance in the medical field.

News

[2023/08/07] 🔥🔥 Released a model instruction-tuned based on Huozi, resulting in a significant improvement in model performance. 🔥🔥

[2023/08/05] The "Bentsao" model was presented as a poster at the CCL 2023 Demo Track.

[2023/08/03] SCIR Lab open-sourced the Huozi general question-answering model. Everyone is welcome to check it out! 🎉🎉

[2023/07/19] Released a model instruction-tuned based on Bloom.

[2023/05/12] The model was renamed from "Huatuo" to "Bentsao".

[2023/04/28] Released a model instruction-tuned based on the Chinese-Alpaca.

[2023/04/24] Released a model instruction-tuned based on LLaMA with medical literature.

[2023/03/31] Released a model instruction-tuned based on LLaMA with a medical knowledge base.

A Quick Start

Firstly, install the required packages. It is recommended to use Python 3.9 or above

text
pip install -r requirements.txt

For all base models, we adopted the semi-precision base model LoRA fine-tuning method for instruction fine-tuning training, in order to strike a balance between computational resources and model performance.

Base models

LoRA weight download

LORA weights can be downloaded through Baidu Netdisk or Huggingface.

  1. 🔥LoRA for Huozi 1.0
  • with the medical knowledge base and medical QA dataset BaiduDisk
  1. LoRA for Bloom
  1. LoRA for Chinese Alpaca
  1. LoRA for LLaMA

Download the LORA weight file and extract it. The format of the extracted file should be as follows:

text
**lora-folder-name**/
  - adapter_config.json   # LoRA configuration
  - adapter_model.bin   # LoRA weight

We also trained a medical version of ChatGLM: ChatGLM-6B-Med based on the same data.

Infer

We provided some test cases in ./data/infer.json, which can be replaced with other datasets. Please make sure to keep the format consistent.

Run the infer script

text
#Based on medical knowledge base
bash ./scripts/infer.sh

#Based on medical literature
#single-epoch
bash ./scripts/infer-literature-single.sh

#multi-epoch
bash ./scripts/infer-literature-multi.sh

The code for the infer.sh script is as follows. Please replace the base model base_model, the LoRA weights lora_weights, and the test dataset path instruct_dir before running

text
python infer.py \
		    --base_model 'BASE_MODEL_PATH' \
		    --lora_weights 'LORA_WEIGHTS_PATH' \
		    --use_lora True \
		    --instruct_dir 'INFER_DATA_PATH' \
		    --prompt_template 'TEMPLATE_PATH'

The prompt template is relevant to the model as follows:

Huozi&Bloom LLaMA&Alpaca
templates/bloom_deploy.json with the medical knowledge basetemplates/med_template.json with the medical literaturetemplates/literature_template.json

other reference in ./scripts/test.sh

Methodology

The base model has limited effectiveness in medical question-answering scenarios. Instruction-tuning is an efficient method to give the base model the ability to answer human questions

Dataset construction

Medical knowledge base

We used both publicly available and self-built Chinese medical knowledge bases, with a primary reference to cMeKG. The medical knowledge base is built around diseases, drugs, and diagnostic indicators, with fields including complications, risk factors, histological examinations, clinical symptoms, drug treatments, and adjuvant therapies. An example of the knowledge base is shown below:

text
{"中心词": "偏头痛", "相关疾病": ["妊娠合并偏头痛", "恶寒发热"], "相关症状": ["皮肤变硬", "头部及眼后部疼痛并能听到连续不断的隆隆声", "晨起头痛加重"], "所属科室": ["中西医结合科", "内科"], "发病部位": ["头部"]}

We used the GPT3.5 API to build a Question-Answer dataset around a medical knowledge base, with multiple prompts to fully leverage the knowledge.

An example of the training dataset for instruct-tuning is shown below:

text
"问题：一位年轻男性长期使用可卡因，突然出现胸痛、呕吐、出汗等症状，经检查发现心电图反映心肌急性损伤，请问可能患的是什么疾病？治疗方式是什么？"
回答: 可能患的是心肌梗塞，需要进行维拉帕米、依普利酮、硝酸甘油、ß阻滞剂、吗啡等药物治疗，并进行溶栓治疗、低分子量肝素、钙通道阻滞剂等辅助治疗。此外需要及时停用可卡因等药物，以防止病情加重。"

We provided a training dataset for the model, consisting of more than eight thousand entries. It should be noted that although knowledge has been incorporated into the construction of the training set, there are still errors and imperfections. We will use better strategies to iteratively update the dataset in the future.

The quality of the dataset for instruct-tuning is still limited. We will continue to iterate and improve it. Meanwhile, the medical knowledge base and dataset construction code are still being organized and will be released once completed.

Medical literature

In addition, we collected Chinese medical literature on liver cancer in 2023, and used the GPT3.5 interface to collect multiple rounds of question-and-answer data around the medical literature. We provide 1k training examples in ./data_literature/liver_cancer.json. At present, the quality of training samples is still limited. In the future, we will further iterate the data and release it in the form of public dataset. An example of a training sample is as follows:

At present, we only open model parameters trained for the single disease "liver cancer". In the future, we plan to release a medical dialogue dataset incorporating medical literature conclusions, and plan to train models for 16 diseases related to "liver, gallbladder and pancreas".

Finetune

To fine-tune LLaMA with your own dataset, please construct your dataset following the format of ./data/llama_data.json and run the finetune script.

text
bash ./scripts/finetune.sh

Training details

GPU resource

We trained the model on an A100-SXM-80GB for a total of 10 epochs, which took approximately 2 hours and 17 minutes. With a batch size of 128, the GPU memory usage was around 40GB. We expect that GPU with 24GB or more of memory, such as the 3090/4090, will be able to support the model well, and the batch size can be adjusted based on the size of the GPU memory.

Experiments

Parameter settings and details training process can be found in the wandb log:
https://wandb.ai/thinksoso/llama_med/runs/a5wgcnzt/overview?workspace=user-thinksoso

Comparison of models

Note: test in March 2023.

Input入 Llama output Alpaca output HuaTuo output
小张最近感觉身体不适，出现心悸、气促等症状。体检发现心脏扩大、搏动减弱。 问题：小张最近感觉身体不适，出现心悸、气促等症状。体检发现心脏扩大、搏动减弱。 最近，小张一直感觉身体不舒服，有心烦意乱、呼吸急促等症状。体格检查显示心脏增大，肌肉无力。 小张可能患有心肌炎，建议尽快到医院进行检查和治疗。治疗方案包括药物治疗和手术治疗。
麻风病和儿童哮喘的病因是否一致？ 儿童哮喘和麻风病的病因是一致，但是儿童哮喘和麻风病的病因不一致 是的，哮喘和儿童喘息有相似的原因。 不一致，麻风病的病因是麻风杆菌感染，而儿童哮喘的病因是喂养不良、吸烟等因素。
一位患者出现了持续性的胀痛和黄疸，经影像学检查后确诊为肝胆管结石病，应该采取哪些治疗方案？ 肝胆管结石病患者应该采取哪些治疗方案？ 患者应接受手术和化疗相结合的治疗。 针对肝胆管结石病，治疗方案包括手术治疗、药物治疗和辅助治疗等。具体治疗方案需要根据患者的具体情况而定。

##FAQs

  1. Q: Why was it renamed to "Bentsao"?

    A: For naming consistency with the SCIR large language model series, the name of the Chinese medical model has been renamed to "Bentsao".

  2. Q: Why "Huatuo(华驼)" instead of "Huatuo(华佗)"?

    A: We chose the name "Huatuo(华驼)" because our base model, LLaMA(美洲驼), and Alpaca(羊驼). Inspired by their names and a pun on the name "tuo驼" & "tuo佗", we named our model "Huatuo(华驼)".

  3. Q: Does it use Chinese traditional medicine theories or data?

    A: Not at the moment.

  4. Q: The results from the model vary and are limited in effectiveness.

    A: Due to the diversity considerations of the generative model, results may vary with multiple runs. The current open-source model, owing to the limited Chinese corpus in LLaMA and Alpaca, and a rather coarse way of knowledge integration, may yield inconsistent results. Please try the bloom-based and character-based models.

  5. Q: The model cannot run or the inferred content is completely unacceptable.

    A: Please ensure that you've installed the dependencies from the requirements, set up the CUDA environment and added the environment variables, and correctly input the downloaded model and Lora's storage location. Any repetition or partial mistakes in the inferred content are occasional issues with the llama-based model and relate to LLaMA's ability in Chinese, training data scale, and hyperparameter settings. Please try the new character-based model. If there are severe issues, please describe in detail the filename, model name, Lora configuration, etc., in an issue. Thank you all.
    Q: Among the released models, which one is the best?
    A: Based on our experience, the character-based model seems to perform relatively better.

Contributors

This project was founded by the Health Intelligence Group of the Research Center for Social Computing and Information Retrieval at Harbin Institute of Technology, including Haochun Wang, Yanrui Du, Chi Liu, Rui Bai, Nuwa Xi, Yuhan Chen, Zewen Qiang, Jianyu Chen, Zijian Li supervised by Associate Professor Sendong Zhao, Professor Bing Qin, and Professor Ting Liu.

Acknowledgements

This project has referred the following open-source projects. We would like to express our gratitude to the developers and researchers involved in those projects.

Disclaimer

The resources related to this project are for academic research purposes only and strictly prohibited for commercial use. When using portions of third-party code, please strictly comply with the corresponding open source licenses. The content generated by the model is influenced by factors such as model computation, randomness, and quantization accuracy loss, and this project cannot guarantee its accuracy. The vast majority of the dataset used in this project is generated by the model, and even if it conforms to certain medical facts, it cannot be used as the basis for actual medical diagnosis. This project does not assume any legal liability for the content output by the model, nor is it responsible for any losses that may be incurred as a result of using the related resources and output results.

Citation

If you use the data or code from this project, please declare the reference:

text
@misc{wang2023huatuo,
      title={HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge}, 
      author={Haochun Wang and Chi Liu and Nuwa Xi and Zewen Qiang and Sendong Zhao and Bing Qin and Ting Liu},
      year={2023},
      eprint={2304.06975},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

File: requirements.txt

accelerate==0.20.1
appdirs==1.4.4
bitsandbytes==0.37.2
black==23.3.0
black[jupyter]==23.3.0
datasets==2.11.0
fire==0.5.0
peft==0.3.0
transformers==4.30.1
gradio==3.33.1
sentencepiece==0.1.97
scipy==1.10.1
wandb

File: benchmark/README.md

评价基准

我们提出了中文医学复杂开放式问题回答基准（CMCOQA），旨在通过在医疗场景中构建复杂的开放式问题来更准确、有效地评估大语言模型的医学水平。

基准构建

我们首先手动构建了100个复杂的医学问题，并通过GPT-4结合Self-Instruct方法扩展至1200个问题，形成CMCOQA基准。并依据ICD-10标准进行问题分类，对某些类别进行调整，以确保数据集涵盖了各种疾病，且分布相对均衡。

评价维度

CMCOQA基准涉及三个评估维度：完整性（Completeness）、深刻性（Depth）和专业性（Professionalism）。在每个维度按照下表的评分标准，使用0到3分进行评分。

维度 标准 描述
完整性 覆盖度 涵盖问题的各个方面，例如病因、诊断、治疗和预防。
相关性 与问题直接相关，避免无关信息。
深刻性 分析深度 提供深入的分析和讨论，而不仅仅是表面的描述。
见解 提供建设性的建议或观点。
专业性 准确性 确保内容准确，避免错误或误导性内容。
简洁性 言简意赅，直奔主题，没有冗余信息。
术语使用 恰当使用专业术语，展现专业水平。

评价结果

我们使用人工和GPT-4o结合的方式，使用CMCOQA基准对一系列医学大模型以及通用领域大模型进行评估，可以看到，在医学大模型领域，基于知识微调的本草大模型表现优异，尤其在回复的深刻性方面，显著优于其他基线。

File: benchmark/question.json

[Binary file]

File: data/infer.json

[Binary file]

File: data/knowledge_tuning_data_sample.txt

[Binary file]

File: templates/bloom_deploy.json

{
"description": "Template used by Med Instruction Tuning",
"prompt_input": "下面是一个问题，运用医学知识来正确回答提问.\n{instruction}\n\n\n### 回答：\n",
"prompt_no_input": "下面是一个问题，运用医学知识来正确回答提问.\n{instruction}\n### 回答：\n",
"response_split": "### 回答："
}

File: templates/literature_template.json

{
"description": "Template used by Alpaca-LoRA.",
"prompt_no_input": "以下是描述任务的说明，编写适当地回复完成请求的响应。\n\n ### 说明:\n{instruction}\n\n### 回复:\n",
"response_split": "### 回复:"
}

File: templates/med_template.json

{
"description": "Template used by Med Instruction Tuning",
"prompt_input": "下面是一个问题，运用医学知识来正确回答提问.\n### 问题:\n{instruction}\n### 回答:\n",
"prompt_no_input": "下面是一个问题，运用医学知识来正确回答提问.\n### 问题:\n{instruction}\n### 回答:\n",
"response_split": "### 回答:"
}

File: utils/README.md

Directory for helpers modules

prompter.py

Prompter class, a template manager.

from utils.prompter import Prompter

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#aidoctor Browse aidoctor catalog → #bloom Browse bloom catalog → #chinese Browse chinese catalog → #huozi Browse huozi catalog → #llama Browse llama catalog → #llm Browse llm catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →