"XrayGLM: Medical Diagnosis of X-ray Chest Film Based on VisualGLM-6B Fine-tuning Training" learning sharing and local project fine-tuning deployment practice

XrayGLM is said to be the first Chinese multi-modal medical model that can look at chest X-rays. I also found time to study it because I paid attention to this recently. By the way, I will share the corresponding records of the learning materials.

The official open source project is provided, the address is here , as follows:

 Recently, large language models (LLMs) in general domains, such as ChatGPT, have achieved remarkable success in following instructions and generating human-like responses, and this success has indirectly promoted the research and development of multimodal large models, such as ChatGPT in general domains. Multimodal large models MiniGPT-4 , mPLUG-Owl , Multimodal-GPT and LLaVA  , however, such multimodal large models rarely appear in research in the medical field, hindering the development of related research. Although visual-med-alpaca has made some very effective work on large medical multimodal models, its data are in English diagnostic reports, which is not conducive to promoting the research and development of large medical multimodal models in the Chinese field. To this end, we developed XrayGLMto solve the above problems. XrayGLMIt shows extraordinary potential in medical imaging diagnosis and multi-round interactive dialogue.

In order to help everyone learn and master, the official also provides corresponding video explanations, the address is here , as follows:

 The project has mainly completed two contributions:

• With the help of ChatGPT and public datasets, we constructed a X光影像-诊断报告pair of medical multimodal datasets;
• We will fine-tune the constructed Chinese chest X-ray diagnostic data set in VisualGLM-6B , and open some training weights for academic research;

The following are the details of the contents of the report:

 The data sets used by the official project are open source data as follows:

• MIMIC-CXR is a publicly available chest radiograph dataset consisting of 377,110 images and 227,827 associated reports.
• OpenI is a dataset of chest radiographs from Indiana University Hospital, including 6,459 images and 3,955 reports.

In the above work, the reported information is unstructured, which is not conducive to scientific research. In order to generate plausible medical reports, we preprocess the two datasets and finally obtain English reports that can be used for training . In addition, in order to better support the development of the Chinese community, with the help of ChatGPT's ability, we translated the English report into Chinese, and finally formed a data set that can be used for training.

 Environment installation and deployment are as follows:

安装环境
# 安装依赖
pip install -r requirements.txt
# 国内换源安装依赖
pip install -i https://mirrors.aliyun.com/pypi/simple/ -r requirements.txt
此时默认会安装deepspeed库（支持sat库训练），此库对于模型推理并非必要，同时部分Windows环境安装此库时会遇到问题。 如果想绕过deepspeed安装，我们可以将命令改为：

# 安装依赖
pip install -i https://mirrors.aliyun.com/pypi/simple/ -r requirements_wo_ds.txt
# 安装SwissArmyTransformer
pip install -i https://mirrors.aliyun.com/pypi/simple/ --no-deps "SwissArmyTransformer>=0.3.6"

The checkpoints-XrayGLM-300 model weight address is here , as follows:

 The checkpoints-XrayGLM-3000 model weight address is here , as follows:

 You can see that the model volume is nearly 16GB.

Running and starting only requires the terminal to execute the following command:

python web_demo.py --from_pretrained checkpoints/checkpoints-XrayGLM-3000

The image diagnosis effect of the example is as follows:

 Example of multiple rounds of dialogue:

 If you are interested, you can experience the training experiment by yourself:

硬件资源
* 实验在A100 (4X, 80GB)上进行

（1）准备诊疗报告(中文)和X光影像在data/Xray文件夹下；
（2）开始训练：
# 设置CUDA变量，主要是为了解决有时候直接训练而出现无法正确加载到显卡问题
export CUDA_VISIBLE_DEVICES=0,1,2,3

# 开始训练
bash finetune_XrayGLM.sh
这里的复现过程非常简单，主要是很多过程我们都为大家准备好了，大家可以随时复现一个自己的XrayGLM。