unsloth微调QwQ32B(4bit)

unsloth微调QwQ32B(4bit)

GPU: 3090 24G

unsloth安装部署

• pip 安装

  pip install unsloth --index https://pypi.mirrors.usrc.edu.cn/simple
  

  source /etc/network_turbopip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git
  

  ​

  ---

注册Wandb以监控模型微调过程

• wandb地址

  https://wandb.ai/site

  ​

• 登录

  下载

  pip install wandb
  

  使用api-key登录

  wandb login
  

  ​

• 使用官网示例看一看

  备注：

  1. 需要联网
  2. 需要将key改为自己的
  3. entity需要提前设立

  import random
  import wandbwandb.login(key="api-key")# Start a new wandb run to track this script.
  run = wandb.init(# Set the wandb entity where your project will be logged (generally your team name).entity="qinchihongye-pa",# Set the wandb project where this run will be logged.project="project_test",# Track hyperparameters and run metadata.config={"learning_rate": 0.02,"architecture": "CNN","dataset": "CIFAR-100","epochs": 10,},
  )# Simulate training.
  epochs = 10
  offset = random.random() / 5
  for epoch in range(2, epochs):acc = 1 - 2**-epoch - random.random() / epoch - offsetloss = 2**-epoch + random.random() / epoch + offset# Log metrics to wandb.run.log({"acc": acc, "loss": loss})# Finish the run and upload any remaining data.
  run.finish()
  

  ​

  ​

---

下载QwQ32B量化模型

• huggingface地址(unsloth量化的4bit，比Q4_K_M量化的损失精度更小)

  https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit

  复制名称

  unsloth/QwQ-32B-unsloth-bnb-4bit

• 假设当前目录为

  /root/lanyun-tmp

• 创建文件夹统一存放Huggingface下载的模型

  mkdir Hugging-Face 
  mkdir -p Hugging-Face/QwQ-32B-unsloth-bnb-4bit
  

• 配置镜像源

  vim ~/.bashrc
  

  填入以下两个，以修改HuggingFace 的镜像源 、模型保存的默认

  export HF_ENDPOINT=https://hf-mirror.com
  export HF_HOME=/root/lanyun-tmp/Hugging-Face

  重新加载，查看环境变量是否生效

  source ~/.bashrcecho $HF_ENDPOINT
  echo $HF_HOME
  

• 安装 HuggingFace 官方下载工具

  pip install -U huggingface_hub
  

• 执行下载模型的命令

  huggingface-cli download --resume-download unsloth/QwQ-32B-unsloth-bnb-4bit --local-dir  /root/lanyun-tmp/Hugging-Face/QwQ-32B-unsloth-bnb-4bitHugging-Face/QwQ-32B-unsloth-bnb-4bit
  

  或者使用python下载

  from huggingface_hub import snapshot_download
  snapshot_download(repo_id = "unsloth/QwQ-32B-unsloth-bnb-4bit",local_dir = "/root/lanyun-tmp/Hugging-Face/QwQ-32B-unsloth-bnb-4bit",
  )
  

  ---

transformers库调用示例

• 代码

  from transformers import AutoModelForCausalLM, AutoTokenizermodel_name = "/root/lanyun-tmp/Hugging-Face/QwQ-32B-unsloth-bnb-4bit"model = AutoModelForCausalLM.from_pretrained(model_name,torch_dtype="auto",device_map="cuda:0",
  )
  tokenizer = AutoTokenizer.from_pretrained(model_name)prompt = "你好"
  messages = [{"role": "user", "content": prompt}
  ]
  text = tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True
  )model_inputs = tokenizer([text], return_tensors="pt").to(model.device)generated_ids = model.generate(**model_inputs,max_new_tokens=32768
  )generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
  ]response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
  print(response)
  

  ​

• 显存占用：23G左右。

  ​

  ---

vllm启动示例

• 启动

  cd /root/lanyun-tmp/Hugging-Facevllm serve ./QwQ-32B-unsloth-bnb-4bit \
  --quantization bitsandbytes \
  --load-format bitsandbytes \
  --max-model-len 500 \
  --port 8081
  

• 调用代码

  from openai import OpenAI
  import openaiopenai.api_key = '1111111' # 这里随便填一个
  openai.base_url = 'http://127.0.0.1:8081/v1'def get_completion(prompt, model="QwQ-32B"):client = OpenAI(api_key=openai.api_key,base_url=openai.base_url)messages = [{"role": "user", "content": prompt}]response = client.chat.completions.create(model=model,messages=messages,stream=False)return response.choices[0].message.contentprompt = '你好，请幽默的介绍下你自己，不少于300字'
  get_completion(prompt, model="./QwQ-32B-unsloth-bnb-4bit")
  

  ---

cot数据集

• FreedomIntelligence/medical-o1-reasoning-SFT

  https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT

• 英文数据集下载

  from datasets import load_dataset
  import rich# Login using e.g. `huggingface-cli login` to access this dataset
  ds = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT", "en")rich.print(ds['train'][0])
  

  ​

• 中文数据集下载

  from datasets import load_dataset
  import rich# Login using e.g. `huggingface-cli login` to access this dataset
  ds = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT", "zh")rich.print(ds['train'][0])
  

  ​

• 下载完成后会看到在HuggingFace目录下的datasets目录中有刚刚下载的数据

  ll /root/lanyun-tmp/Hugging-Face/datasets/
  

  ​

  ---

unsloth加载QwQ32b模型

• unsloth支持直接加载模型并推理,先加载模型

  from unsloth import FastLanguageModelmax_seq_length = 2048
  dtype = None
  load_in_4bit = True # 4bitmodel,tokenizer = FastLanguageModel.from_pretrained(model_name = "/root/lanyun-tmp/Hugging-Face/QwQ-32B-unsloth-bnb-4bit/",max_seq_length = max_seq_length,dtype = dtype,load_in_4bit = load_in_4bit,
  )
  

  ​

  显存占用22G左右

  ​

• 推理

  # 将模型调整为推理模式
  FastLanguageModel.for_inference(model)def QwQ32b_infer(question):# prompt模板prompt_style_chat = """请写出一个恰当的回来来完成当前对话任务。### Instruction:你是一名助人为乐的助手。### Question:{}### Response:<think>{}"""# [prompt_style_chat.format(question,"")]inputs = tokenizer([prompt_style_chat.format(question, "")],return_tensors="pt").to("cuda")outputs = model.generate(input_ids = inputs.input_ids,max_new_tokens=2048,use_cache=True,)response = tokenizer.batch_decode(outputs)return response[0].split("### Response:")[1]question = "证明根号2是无理数"
  response = QwQ32b_infer(question)
  

  ​

  ---

模型微调

• 测试:使用微调数据集进行测试

  question_1 = "根据描述，一个1岁的孩子在夏季头皮出现多处小结节，长期不愈合，且现在疮大如梅，溃破流脓，口不收敛，头皮下有空洞，患处皮肤增厚。这种病症在中医中诊断为什么病？"question_2 = "一个生后8天的男婴因皮肤黄染伴发热和拒乳入院。体检发现其皮肤明显黄染，肝脾肿大和脐部少量渗液伴脐周红肿。在此情况下，哪种检查方法最有助于确诊感染病因？"response_1 = QwQ32b_infer(question_1)
  response_2 = QwQ32b_infer(question_2)print(response_1)
  print(response_2)
  

  ​

  ​

• 加载并处理数据，选择训练集前500条进行最小可行性实验

  import os
  from datasets import load_dataset# 问答提示词模板
  train_prompt_style = """下面是描述任务的指令，与提供进一步上下文的输入配对。编写适当完成请求的响应。在回答之前，仔细思考问题，并创建逐步的思想链，以确保逻辑和准确的响应。### Instruction:
  您是一位在临床推理、诊断和治疗计划方面拥有先进知识的医学专家。请回答以下医学问题。 ### Question:
  {}### Response:
  <think>
  {}
  </think>
  {}"""# 文本生成结束的基本标记
  EOS_TOKEN = tokenizer.eos_token
  tokenizer.eos_token # '<|im_end|>'# 定义函数，对数据集进行修改
  def formatting_prompts_func(examples):inputs = examples["Question"]cots = examples["Complex_CoT"]outputs = examples["Response"]texts = []for input, cot, output in zip(inputs, cots, outputs):text = train_prompt_style.format(input, cot, output) + EOS_TOKENtexts.append(text)return {"text": texts,}# 先选择训练集前500条数据
  dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT","zh", split = "train[0:500]",trust_remote_code=True)
  dataset = dataset.map(formatting_prompts_func, batched = True)import rich
  rich.print(dataset[0])
  rich.print(dataset[0]['text'])
  

  ​

• 将模型设置为微调模式

                         
  # 将模型设置为微调模式
  model = FastLanguageModel.get_peft_model(model,r=4, # r=16 # 低秩矩阵的秩target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj",],lora_alpha=16,lora_dropout=0,  bias="none",  use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long contextrandom_state=1024,use_rslora=False,  loftq_config=None,
  )
  

  ​

• 创建训练器（有监督微调对象）

  from trl import SFTTrainer
  from transformers import TrainingArguments
  from unsloth import is_bfloat16_supportedtrainer = SFTTrainer(model=model, # 指定需要微调的预训练模型tokenizer=tokenizer, # 分词器train_dataset=dataset, # 训练数据dataset_text_field="text", # 指定数据集中那一列包含训练文本（在formatting_prompt_func里面指定）max_seq_length=max_seq_length, #最大序列长度，用于控制输入文本的最大token数量dataset_num_proc=2, # 数据加载的并行进程数args=TrainingArguments(per_device_train_batch_size=1, # 每个GPU/设备的戌年批量大小（较小值适合大模型）gradient_accumulation_steps=4, # 梯度累计步数，相当于batch_size=1*4=4# num_train_epochs = 1, # 如果设置了num_train_epochs，则max_steps失效warmup_steps=5, # 预热步数，初始阶段学习率较低，然后逐步升高max_steps=60,# 最大训练步数learning_rate=2e-4, # 学习率fp16=not is_bfloat16_supported(),  # 如果GPU不支持bfloat16，则使用fp16（16位浮点数）bf16=is_bfloat16_supported(), # 如果GPU支持bfloat16，则启用bf16（训练更稳定）logging_steps=10, # 每10步记录一次日志optim="adamw_8bit", # 使用adamw_8bit 8bit adamw优化器减少显存占用weight_decay=0.01, # 权重衰减 L2正则化，防止过拟合lr_scheduler_type="linear", # 学习率调整策略，线性衰减seed=1024, # 随机种子，保证实验结果可复现output_dir="/root/lanyun-tmp/outputs", # 训练结果的输出目录),
  )# 设置wandb(可选则)
  import wandb
  wandb.login(key="api-key")run = wandb.init(entity="qinchihongye-pa",project='QwQ-32B-4bit-FT')# 开始模型微调
  trainer_stats = trainer.train()trainer_status
  

  ​

  训练过程中的显存占用如上，训练过程如下

  ​

  点击wandb链接，查看训练过程中的损失函数，学习率，梯度等等的变化。

  ​

• unsloth在微调结束后，会自动更新模型权重（在缓存中），因此无序手动合并集合直接调用微调后的模型

  FastLanguageModel.for_inference(model)new_response_1 = QwQ32b_infer(question_1)
  new_response_2 = QwQ32b_infer(question_2)new_response_1
  new_response_2
  

  ​

  ​

  可以看到第一个问题还是回答错了，第二个问题也如旧，可以考虑继续进行大规模微调，使用全部微调文件+多个epoch。

• 模型合并

  此时本地保存的模型权重在/root/lanyun-tmp/outputs中

  ​

  注意，unsloth中默认100步保存一个checkpoint，因为当前steps=60,所以只有一个checkpoint点。

  合并保存为safetensors

  model.save_pretrained_merged("/root/lanyun-tmp/QwQ-Medical-COT-Tiny", tokenizer, save_method = "merged_4bit_forced",#保存为4bit量化)# model.save_pretrained_merged("dir"
  #                              , tokenizer
  #                              , save_method = "merged_16bit",#保存为16bit
  #                             )
  

  合并为GGUF格式（需要量化，非常耗时）

  # model.save_pretrained_gguf("dir"
  #                            , tokenizer
  #                            , quantization_method = "q4_k_m"
  #                           )# model.save_pretrained_gguf("dir"
  #                            , tokenizer
  #                            , quantization_method = "q8_0"
  #                           )# model.save_pretrained_gguf("dir"
  #                            , tokenizer
  #                            , quantization_method = "f16"
  #                           )
  

  ---