423A35C7/LLaMA-Factory
1
0
Fork 0
mirror of https://github.com/hiyouga/LLaMA-Factory.git synced 2025-12-15 03:10:35 +08:00
Code Issues Packages Projects Releases Wiki Activity

Added Mixture of Depths

Browse Source
This commit is contained in:
Marco
2024-04-18 20:31:24 +02:00
parent 2aaaede247
commit 620add7b9f
10 changed files with 103 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
examples/README.md
View File

@@ -41,6 +41,9 @@ examples/
├── llama_pro/
│ ├── expand.sh: Expand layers in the model
│ └── sft.sh: Fine-tune the expanded model
├── MoD/
│ ├── freeze_sft.sh: Freeze finetune a model, updating only the MoD router
│ └── sft.sh: Fine-tune the MoD model
└── fsdp_qlora/
└── sft.sh: Fine-tune quantized model with FSDP+QLoRA
```