mirror of
https://github.com/hiyouga/LLaMA-Factory.git
synced 2026-08-17 13:35:44 +08:00
84 lines
2.9 KiB
Markdown
84 lines
2.9 KiB
Markdown
# KTransformers LoRA SFT
|
|
|
|
KTransformers (KT) executes routed MoE experts on CPU while LLaMA-Factory remains responsible for data,
|
|
LoRA arguments, and the training entry point. The production scope is routed-BF16 and routed-INT8 LoRA.
|
|
|
|
KT has one user configuration source: the training YAML. Accelerate YAML contains FSDP2 settings only.
|
|
LLaMA-Factory derives LoRA rank, alpha, dropout, activation policy, and local runtime capacity.
|
|
|
|
```yaml
|
|
finetuning_type: lora
|
|
lora_rank: 8
|
|
lora_alpha: 16
|
|
lora_target: all
|
|
|
|
use_kt: true
|
|
disable_gradient_checkpointing: false
|
|
kt_cpu_activation: retain
|
|
kt_config:
|
|
kt_expert_weight_format: bf16
|
|
kt_backend: AMXBF16
|
|
kt_num_threads: 96
|
|
kt_tp_enabled: true
|
|
kt_threadpool_count: 2
|
|
kt_max_cache_depth: 2
|
|
```
|
|
|
|
Routed INT8 additionally requires matching expert and BF16 non-expert artifacts:
|
|
|
|
```yaml
|
|
kt_weight_path: /abs/path/to/routed-int8-experts
|
|
kt_non_expert_weight_path: /abs/path/to/bf16-non-expert-cache
|
|
kt_config:
|
|
kt_expert_weight_format: int8
|
|
kt_backend: auto
|
|
kt_weight_lifecycle: persistent
|
|
```
|
|
|
|
Launch the standard training entry point through Accelerate:
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0,1 accelerate launch \
|
|
--config_file examples/ktransformers/accelerate/fsdp2_kt_bf16.yaml \
|
|
src/train.py examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml
|
|
```
|
|
|
|
## Load a saved adapter
|
|
|
|
Use a local, complete KT adapter directory for chat or evaluation. Repeat the training LoRA shape (`finetuning_type`,
|
|
`lora_rank`, `lora_alpha`, and `lora_dropout`) and the KT base-weight settings. In particular, routed INT8 loading
|
|
must use the same `kt_weight_path` and `kt_non_expert_weight_path` as training.
|
|
|
|
```yaml
|
|
model_name_or_path: /abs/path/to/base-model
|
|
adapter_name_or_path: /abs/path/to/output/checkpoint-300
|
|
finetuning_type: lora
|
|
lora_rank: 8
|
|
lora_alpha: 16
|
|
lora_dropout: 0.0
|
|
|
|
use_kt: true
|
|
kt_cpu_activation: retain
|
|
kt_config:
|
|
kt_expert_weight_format: bf16
|
|
kt_backend: AMXBF16
|
|
kt_num_threads: 96
|
|
```
|
|
|
|
```bash
|
|
llamafactory-cli chat path/to/kt_adapter_infer.yaml
|
|
llamafactory-cli eval path/to/kt_adapter_eval.yaml
|
|
```
|
|
|
|
The directory must contain the standard PEFT adapter files and, when fused routed-expert LoRA is used,
|
|
`fused_expert_lora.safetensors` plus `kt_adapter_manifest.json`. LLaMA-Factory first loads the standard PEFT
|
|
adapter, then KT validates and restores the fused artifact. `adapter_folder` may select a local subdirectory;
|
|
paths outside the adapter root and Hub adapter IDs fail before model loading. Download a Hub bundle locally first.
|
|
|
|
For training resume, keep the original training YAML and use `resume_from_checkpoint`. The optimizer checkpoint
|
|
currently requires the same distributed world size. Missing, tampered, or mismatched artifacts fail closed instead
|
|
of falling back to the source checkpoint.
|
|
|
|
Do not combine KT with a second Transformers/FSDP checkpoint wrapper or Unsloth GC, and do not put `kt_config`
|
|
in the Accelerate YAML. See the BF16 and INT8 examples under `examples/ktransformers/train_lora/`.
|