LLaMA-Factory/docs/en/advanced/ktransformers.md

KTransformers LoRA SFT

KTransformers (KT) executes routed MoE experts on CPU while LLaMA-Factory remains responsible for data, LoRA arguments, and the training entry point. The production scope is routed-BF16 and routed-INT8 LoRA.

KT has one user configuration source: the training YAML. Accelerate YAML contains FSDP2 settings only. LLaMA-Factory derives LoRA rank, alpha, dropout, activation policy, and local runtime capacity.

finetuning_type: lora
lora_rank: 8
lora_alpha: 16
lora_target: all

use_kt: true
disable_gradient_checkpointing: false
kt_cpu_activation: retain
kt_config:
  kt_expert_weight_format: bf16
  kt_backend: AMXBF16
  kt_num_threads: 96
  kt_tp_enabled: true
  kt_threadpool_count: 2
  kt_max_cache_depth: 2

Routed INT8 additionally requires matching expert and BF16 non-expert artifacts:

kt_weight_path: /abs/path/to/routed-int8-experts
kt_non_expert_weight_path: /abs/path/to/bf16-non-expert-cache
kt_config:
  kt_expert_weight_format: int8
  kt_backend: auto
  kt_weight_lifecycle: persistent

Launch the standard training entry point through Accelerate:

CUDA_VISIBLE_DEVICES=0,1 accelerate launch \
  --config_file examples/ktransformers/accelerate/fsdp2_kt_bf16.yaml \
  src/train.py examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml

Load a saved adapter

Use a local, complete KT adapter directory for chat or evaluation. Repeat the training LoRA shape (finetuning_type, lora_rank, lora_alpha, and lora_dropout) and the KT base-weight settings. In particular, routed INT8 loading must use the same kt_weight_path and kt_non_expert_weight_path as training.

model_name_or_path: /abs/path/to/base-model
adapter_name_or_path: /abs/path/to/output/checkpoint-300
finetuning_type: lora
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.0

use_kt: true
kt_cpu_activation: retain
kt_config:
  kt_expert_weight_format: bf16
  kt_backend: AMXBF16
  kt_num_threads: 96

llamafactory-cli chat path/to/kt_adapter_infer.yaml
llamafactory-cli eval path/to/kt_adapter_eval.yaml

The directory must contain the standard PEFT adapter files and, when fused routed-expert LoRA is used, fused_expert_lora.safetensors plus kt_adapter_manifest.json. LLaMA-Factory first loads the standard PEFT adapter, then KT validates and restores the fused artifact. adapter_folder may select a local subdirectory; paths outside the adapter root and Hub adapter IDs fail before model loading. Download a Hub bundle locally first.

For training resume, keep the original training YAML and use resume_from_checkpoint. The optimizer checkpoint currently requires the same distributed world size. Missing, tampered, or mismatched artifacts fail closed instead of falling back to the source checkpoint.

Do not combine KT with a second Transformers/FSDP checkpoint wrapper or Unsloth GC, and do not put kt_config in the Accelerate YAML. See the BF16 and INT8 examples under examples/ktransformers/train_lora/.