mirror of
https://github.com/hiyouga/LLaMA-Factory.git
synced 2025-07-31 10:42:50 +08:00
LLaMA Efficient Tuning
- Download the weights of the LLaMA models.
- Convert them to HF format using this script
python convert_llama_weights_to_hf.py \
--input_dir path_to_llama_weights --model_size 7B --output_dir llama_7b
- Fine-tune the LLaMA models.
CUDA_VISIBLE_DEVICES=0 python src/train_sft.py \
--model_name_or_path llama_7b \
--do_train \
--dataset alpaca_gpt4_zh \
--finetuning_type lora \
--output_dir path_to_sft_checkpoint \
--overwrite_cache \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 2 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 100 \
--learning_rate 1e-5 \
--num_train_epochs 1.0 \
--fp16
Description
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
agentaideepseekfine-tuninggemmagptinstruction-tuninglarge-language-modelsllamallama3llmloramoenlppeftqloraquantizationqwenrlhftransformers
Readme
Apache-2.0
1.4 GiB
Languages
Python
99.6%
Dockerfile
0.4%