423A35C7/LLaMA-Factory
1
0
Fork 0
mirror of https://github.com/hiyouga/LLaMA-Factory.git synced 2025-11-05 02:12:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
LLaMA-Factory/examples/merge_lora
History
hiyouga b12176d818 simplify readme 
Former-commit-id: 0da6ec2d516326fe9c7583ba71cd1778eb838178
2024-04-02 20:07:43 +08:00
..
merge.sh
simplify readme
2024-04-02 20:07:43 +08:00
quantize.sh
support vllm
2024-03-07 20:26:31 +08:00
README.md
simplify readme
2024-04-02 20:07:43 +08:00

README.md

Warning

Merging LoRA weights into a quantized model is not supported.

Tip

Use --model_name_or_path path_to_model solely to use the exported model or model fine-tuned in full/freeze mode.

Use CUDA_VISIBLE_DEVICES=0, --export_quantization_bit 4 and --export_quantization_dataset data/c4_demo.json to quantize the model with AutoGPTQ after merging the LoRA weights.

Usage:

  • merge.sh: merge the lora weights
  • quantize.sh: quantize the model with AutoGPTQ (must after merge.sh, optional)