423A35C7/LLaMA-Factory
1
0
Fork 0
mirror of https://github.com/hiyouga/LLaMA-Factory.git synced 2025-12-15 19:30:36 +08:00
Code Issues Packages Projects Releases Wiki Activity

support layerwise galore

Browse Source
This commit is contained in:
hiyouga
2024-03-10 00:24:11 +08:00
parent 18ffce36b5
commit 8664262cde
14 changed files with 109 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
examples/extras/galore/adamw_8bit_bf16.sh
View File

@@ -16,7 +16,7 @@ CUDA_VISIBLE_DEVICES=0 python ../../../src/train_bash.py \
--preprocessing_num_workers 16 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 2 \
--gradient_accumulation_steps 1 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--warmup_steps 20 \