LLM微调(二)| 微调LLAMA-2和其他开源LLM的两种简单方法
本文将介绍两种开源工具来微调LLAMA-2。
一、使用autotrain-advanced微调LLAMA-2
? ? ? ? AutoTrain是一种无代码工具,用于为自然语言处理(NLP)任务、计算机视觉(CV)任务、语音任务甚至表格任务训练最先进的模型。
1) 安装相关库,使用huggingface_hub下载微调数据
!pip install autotrain-advanced
!pip install huggingface_hub
2) 更新autotrain-advanced所需要的包
# update torch
!autotrain?setup?--update-torch
3) 登录Huggingface
# Login to huggingface
?
from huggingface_hub import notebook_login
notebook_login()
4)?开始微调LLAMA-2
! autotrain llm \
--train \
--model {MODEL_NAME} \
--project-name {PROJECT_NAME} \
--data-path data/ \
--text-column text \
--lr {LEARNING_RATE} \
--batch-size {BATCH_SIZE} \
--epochs {NUM_EPOCHS} \
--block-size {BLOCK_SIZE} \
--warmup-ratio {WARMUP_RATIO} \
--lora-r {LORA_R} \
--lora-alpha {LORA_ALPHA} \
--lora-dropout {LORA_DROPOUT} \
--weight-decay {WEIGHT_DECAY} \
--gradient-accumulation {GRADIENT_ACCUMULATION}
核心参数含义:
llm: 微调模型的类型
— project_name:?项目名称
— model:?需要微调的基础模型
— data_path: 指定微调所需要的数据,可以使用huggingface上的数据集
— text_column: 如果数据是表格,需要指定instructions和responses对应的列名
— use_peft: 指定peft某一种方法
— use_int4:?指定int 4量化
— learning_rate:?学习率
— train_batch_size: 训练批次大小
— num_train_epochs: 训练轮数大小
— trainer:?指定训练的方式
— model_max_length: 设置模型最大上下文窗口
— push_to_hub(可选): 微调好的模型是否需要存储到Hugging Face??
— repo_id: 如果要存储微调好的模型到Hugging Face,需要指定repository ID
— block_size:?设置文本块大小
下面看一个具体的示例:
!autotrain llm
--train
--project_name "llama2-autotrain-openassitant"
--model TinyPixel/Llama-2-7B-bf16-sharded
--data_path timdettmers/openassistant-guanaco
--text_column text
--use_peft
--use_int4
--learning_rate 0.4
--train_batch_size 3
--num_train_epochs 2
--trainer sft
--model_max_length 1048
--push_to_hub
--repo_id trojrobert/llama2-autotrain-openassistant
--block_size 1048 > training.log
二、使用TRL微调LLAMA-2
? ? ? ?TRL是一个全栈库,提供了通过强化学习来训练transformer语言模型一系列工具,包括从监督微调步骤(SFT)、奖励建模步骤(RM)到近端策略优化(PPO)步骤。
1)安装相关的库
!pip install -q -U trl peft transformers datasets bitsandbytes wandb
2)从Huggingface导入数据集
from datasets import load_dataset
?
dataset_name = "timdettmers/openassistant-guanaco"
?
dataset = load_dataset(dataset_name, split="train")
3)量化配置,从Huggingface下载模型
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
?
?
# quantizition configuration
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
)
?
?
# download model
?
model_name = "TinyPixel/Llama-2-7B-bf16-sharded"
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
trust_remote_code=True
)
model.config.use_cache = False
4)下载Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
5)创建PEFT配置
from peft import LoraConfig, get_peft_model
?
lora_alpha = 16
lora_dropout = 0.1
lora_r = 64
?
peft_config = LoraConfig(
lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
r=lora_r,
bias="none",
task_type="CAUSAL_LM"
)
6)创建微调和训练配置
from transformers import TrainingArguments
?
output_dir = "./results"
per_device_train_batch_size = 4
gradient_accumulation_steps = 4
optim = "paged_adamw_32bit"
save_steps = 100
logging_steps = 10
learning_rate = 2e-4
max_grad_norm = 0.3
max_steps = 100
warmup_ratio = 0.03
lr_scheduler_type = "constant"
?
training_arguments = TrainingArguments(
output_dir=output_dir,
per_device_train_batch_size=per_device_train_batch_size,
gradient_accumulation_steps=gradient_accumulation_steps,
optim=optim,
save_steps=save_steps,
logging_steps=logging_steps,
learning_rate=learning_rate,
fp16=True,
max_grad_norm=max_grad_norm,
max_steps=max_steps,
warmup_ratio=warmup_ratio,
group_by_length=True,
lr_scheduler_type=lr_scheduler_type,
)
7)创建SFTTrainer配置
from trl import SFTTrainer
?
max_seq_length = 512
?
trainer = SFTTrainer(
model=model,
train_dataset=dataset,
peft_config=peft_config,
dataset_text_field="text",
max_seq_length=max_seq_length,
tokenizer=tokenizer,
args=training_arguments,
)
8)在微调的时候,对LN层使用float 32训练更稳定
for name, module in trainer.model.named_modules():
if "norm" in name:
module = module.to(torch.float32)
9)开始微调
trainer.train()
10)保存微调好的模型
model_to_save = trainer.model.module if hasattr(trainer.model, 'module') else trainer.model # Take care of distributed/parallel training
model_to_save.save_pretrained("outputs")
11)加载微调好的模型
lora_config = LoraConfig.from_pretrained('outputs')
tuned_model = get_peft_model(model, lora_config)
12)测试微调好的模型效果
?
text = "What is a large language model?"
device = "cuda:0"
?
inputs = tokenizer(text, return_tensors="pt").to(device)
outputs = tuned_model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
参考文献:
[1]?https://trojrobert.medium.com/4-easier-ways-for-fine-tuning-llama-2-and-other-open-source-llms-eb3218657f6e
[2]?https://colab.research.google.com/drive/1JMEi2VMNGMOTyfEcQZyp23EISUrWg5cg?usp=sharing
[3]?https://colab.research.google.com/drive/1ctevXhrE60s7o9RzsxpIqq37EjyU9tBn?usp=sharing#scrollTo=bsbdrb5p2ONa
本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符,请联系我的编程经验分享网邮箱:veading@qq.com进行投诉反馈,一经查实,立即删除!