本文将介绍两种开源工具来微调LLAMA-2。
一、使用autotrain-advanced微调LLAMA-2
AutoTrain是一种无代码工具，用于为自然语言处理（NLP）任务、计算机视觉（CV）任务、语音任务甚至表格任务训练最先进的模型。
1） 安装相关库，使用huggingface_hub下载微调数据
!pip install autotrain-advanced!pip install huggingface_hub
2） 更新autotrain-advanced所需要的包
# update torch!autotrain setup --update-torch
3） 登录Huggingface
# Login to huggingfacefrom huggingface_hub import notebook_loginnotebook_login()
4） 开始微调LLAMA-2
! autotrain llm \--train \--model {MODEL_NAME} \--project-name {PROJECT_NAME} \--data-path data/ \--text-column text \--lr {LEARNING_RATE} \--batch-size {BATCH_SIZE} \--epochs {NUM_EPOCHS} \--block-size {BLOCK_SIZE} \--warmup-ratio {WARMUP_RATIO} \--lora-r {LORA_R} \--lora-alpha {LORA_ALPHA} \--lora-dropout {LORA_DROPOUT} \--weight-decay {WEIGHT_DECAY} \--gradient-accumulation {GRADIENT_ACCUMULATION}
核心参数含义：
llm: 微调模型的类型
— project_name: 项目名称
— model: 需要微调的基础模型
— data_path: 指定微调所需要的数据，可以使用huggingface上的数据集
— text_column: 如果数据是表格，需要指定instructions和responses对应的列名
— use_peft: 指定peft某一种方法
— use_int4: 指定int 4量化
— learning_rate: 学习率
— train_batch_size: 训练批次大小
— num_train_epochs: 训练轮数大小
— trainer: 指定训练的方式
— model_max_length: 设置模型最大上下文窗口
— push_to_hub（可选）: 微调好的模型是否需要存储到Hugging Face?
— repo_id: 如果要存储微调好的模型到Hugging Face，需要指定repository ID
— block_size: 设置文本块大小
下面看一个具体的示例：
!autotrain llm--train--project_name "llama2-autotrain-openassitant"--model TinyPixel/Llama-2-7B-bf16-sharded--data_path timdettmers/openassistant-guanaco--text_column text--use_peft--use_int4--learning_rate 0.4--train_batch_size 3--num_train_epochs 2--trainer sft--model_max_length 1048--push_to_hub--repo_id trojrobert/llama2-autotrain-openassistant--block_size 1048 > training.log
二、使用TRL微调LLAMA-2
TRL是一个全栈库，提供了通过强化学习来训练transformer语言模型一系列工具，包括从监督微调步骤（SFT）、奖励建模步骤（RM）到近端策略优化（PPO）步骤。
1）安装相关的库
!pip install -q -U trl peft transformers datasets bitsandbytes wandb2）从Huggingface导入数据集
from datasets import load_datasetdataset_name = "timdettmers/openassistant-guanaco"dataset = load_dataset(dataset_name, split="train")
3）量化配置，从Huggingface下载模型
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig# quantizition configurationbnb_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type="nf4",bnb_4bit_compute_dtype=torch.float16,)# download modelmodel_name = "TinyPixel/Llama-2-7B-bf16-sharded"model = AutoModelForCausalLM.from_pretrained(model_name,quantization_config=bnb_config,trust_remote_code=True)model.config.use_cache = False
4）下载Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)tokenizer.pad_token = tokenizer.eos_token
5）创建PEFT配置
from peft import LoraConfig, get_peft_modellora_alpha = 16lora_dropout = 0.1lora_r = 64peft_config = LoraConfig(lora_alpha=lora_alpha,lora_dropout=lora_dropout,r=lora_r,bias="none",task_type="CAUSAL_LM")
6）创建微调和训练配置
from transformers import TrainingArgumentsoutput_dir = "./results"per_device_train_batch_size = 4gradient_accumulation_steps = 4optim = "paged_adamw_32bit"save_steps = 100logging_steps = 10learning_rate = 2e-4max_grad_norm = 0.3max_steps = 100warmup_ratio = 0.03lr_scheduler_type = "constant"training_arguments = TrainingArguments(output_dir=output_dir,per_device_train_batch_size=per_device_train_batch_size,gradient_accumulation_steps=gradient_accumulation_steps,optim=optim,save_steps=save_steps,logging_steps=logging_steps,learning_rate=learning_rate,fp16=True,max_grad_norm=max_grad_norm,max_steps=max_steps,warmup_ratio=warmup_ratio,group_by_length=True,lr_scheduler_type=lr_scheduler_type,)
7）创建SFTTrainer配置
from trl import SFTTrainermax_seq_length = 512trainer = SFTTrainer(model=model,train_dataset=dataset,peft_config=peft_config,dataset_text_field="text",max_seq_length=max_seq_length,tokenizer=tokenizer,args=training_arguments,)
8）在微调的时候，对LN层使用float 32训练更稳定
for name, module in trainer.model.named_modules():if "norm" in name:module = module.to(torch.float32)
9）开始微调
trainer.train()10）保存微调好的模型
model_to_save = trainer.model.module if hasattr(trainer.model, 'module') else trainer.model # Take care of distributed/parallel trainingmodel_to_save.save_pretrained("outputs")
11）加载微调好的模型
lora_config = LoraConfig.from_pretrained('outputs')tuned_model = get_peft_model(model, lora_config)
12）测试微调好的模型效果
text = "What is a large language model?"device = "cuda:0"inputs = tokenizer(text, return_tensors="pt").to(device)outputs = tuned_model.generate(**inputs, max_new_tokens=50)print(tokenizer.decode(outputs[0], skip_special_tokens=True))
参考文献：
[1] https://trojrobert.medium.com/4-easier-ways-for-fine-tuning-llama-2-and-other-open-source-llms-eb3218657f6e
[2] https://colab.research.google.com/drive/1JMEi2VMNGMOTyfEcQZyp23EISUrWg5cg?usp=sharing
[3] https://colab.research.google.com/drive/1ctevXhrE60s7o9RzsxpIqq37EjyU9tBn?usp=sharing#scrollTo=bsbdrb5p2ONa