423A35C7/LLaMA-Factory
1
0
Fork 0
mirror of https://github.com/hiyouga/LLaMA-Factory.git synced 2026-11-06 06:59:59 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: add Qwen3.8 LlamaFactory SFT skill (#10901)

Browse Source 
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Cui-yshoho
2026-11-04 20:39:18 +08:00
committed by GitHub
parent a3e7f60ae6
commit 84024cefa6
12 changed files with 702 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
.agents/skills/llamafactory-qwen38-sft/SKILL.md Normal file
View File

@@ -0,0 +1,49 @@
---
name: llamafactory-qwen38-sft
description: Run end-to-end LlamaFactory SFT for Qwen/Qwen3.8-27B using single-device 4-bit QLoRA or multi-device HyperParallel FSDP2 full tuning, including setup, training validation, artifact evaluation, export, and local serving. Use only for Qwen3.8-27B.
---
# Qwen3.8-27B SFT with LlamaFactory
Drive LlamaFactory directly from environment setup through a verified local artifact. Do not build a separate agent runtime or training harness.
## Model contract
- Use the exact model `Qwen/Qwen3.8-27B`; do not substitute Qwen3-8B or Qwen3.5-27B.
- Use the LlamaFactory template `qwen3_8`. Transformers correctly exposes the architecture as `qwen3_5` / `Qwen3_5ForConditionalGeneration`.
- Treat the model as multimodal and choose text-only or multimodal data deliberately. Preserve the repository example's model-component defaults unless the requested tuning scope requires an override.
- Keep the `qwen3_8` template when changing thinking behavior. Record explicit `enable_thinking` or `reasoning_effort` overrides with the run configuration.
- Use single-device 4-bit QLoRA or multi-device HyperParallel with Accelerate FSDP2 full tuning. The checked-in HyperParallel recipe is smoke-tested on Ascend; treat CUDA HyperParallel and artifact-level completion as pending until they independently pass the applicable acceptance gates. Do not silently replace either path when a dependency or capacity gate fails.
## References
Load only the material needed for the requested stage:
- For environment creation, dependency gates, hardware selection, or model-cache validation, read [setup.md](references/setup.md).
- For the Ascend 4-bit QLoRA dependency path, additionally read [ascend-qlora.md](references/ascend-qlora.md).
- For training configuration, QLoRA, HyperParallel, or the training smoke test, read [training.md](references/training.md).
- For evaluation, artifact validation, export, inference, or local API verification, read [serving.md](references/serving.md).
- For the complete workflow, read them in that order.
## Workflow
1. Establish the requested scope, dataset, accelerator, selected devices, and time and disk limits.
2. Record the LlamaFactory revision and validate the environment and local model before training.
3. Validate and register custom data using the current repository documentation; preserve source data and keep held-out samples separate.
4. Resolve the training parameters and keep generated configuration, logs, and artifacts in explicit run-specific directories.
5. Run the smallest representative smoke test at the requested validation scope, then use a fresh output directory for a formal run.
6. When an artifact is requested, reload it before export or serving. Compare the base and tuned models with the same held-out prompts and decoding settings.
7. Report versions, hardware, configs, commands, metrics, artifact paths, failures, fixes, and untested limitations. Mark conclusions as code-confirmed, server-validated, or pending.
## Execution boundaries
- Pin the selected devices with the platform visibility variable and do not share occupied devices without explicit agreement.
- Run long downloads and training jobs with durable logs. Verify the real worker process and output artifacts; a detached launcher exiting successfully is not completion.
- Keep base weights, generated configs and logs, and training artifacts in separate paths. Record their resolved absolute paths.
- Bind services to `127.0.0.1` by default. External exposure requires explicit authorization, authentication, and a network plan.
- Never print or persist credentials.
- Do not alter model code merely to make the workflow pass. Preserve reproducible framework failures and treat their fixes as separately testable changes.
## Package maintenance
Treat `.agents/skills/llamafactory-qwen38-sft` as the canonical package. When changing it, update the `.claude/skills/llamafactory-qwen38-sft` compatibility mirror byte-for-byte and verify both directories with `diff -qr` before committing. This rule is for skill maintenance and does not add a step to an SFT run.

15
.agents/skills/llamafactory-qwen38-sft/references/ascend-qlora.md Normal file
View File

@@ -0,0 +1,15 @@
# Ascend 4-bit QLoRA dependency path
Read this reference only for single-device 4-bit QLoRA on Ascend. The LlamaFactory checkout's current README and `examples/train_qlora/qwen3_lora_sft_bnb_npu.yaml` remain authoritative.
Build the NPU backend from the `multi-backend-refactor` branch of `bitsandbytes-foundation/bitsandbytes` with CMake `COMPUTE_BACKEND=npu`; a regular CUDA-oriented bitsandbytes wheel is not an Ascend acceptance result.
Prefer building a wheel in an isolated build environment and installing that wheel into the training environment with `--no-deps`. Record the bitsandbytes commit and wheel checksum so the environment can be reproduced without letting the build toolchain replace the selected Python or PyTorch stack.
Before training:
1. Import bitsandbytes from the selected training environment and confirm that the NPU shared library, rather than a CUDA fallback, is loaded.
2. Run a small NPU tensor-quantization operation. Import success alone is insufficient.
3. Preserve `double_quantization: false` from the repository's NPU example unless a separately validated backend revision supports otherwise.
If import or the backend operation fails, preserve the build and runtime logs before troubleshooting. Inspect `libbitsandbytes_npu.so` with `readelf` and `ldd`, confirm that its dependencies resolve, and check whether its RPATH or RUNPATH refers to a removed build environment. Stop without replacing the requested QLoRA path with another tuning method when the backend gate remains unresolved.

137
.agents/skills/llamafactory-qwen38-sft/references/serving.md Normal file
View File

@@ -0,0 +1,137 @@
# Artifact evaluation and serving
Use this reference only after a training artifact exists and passes the reload requirement in [training.md](training.md).
## Identify the artifact
- A QLoRA run produces an adapter. Keep the original base model and load it with `adapter_name_or_path`.
- A HyperParallel full-SFT run produces a directly loadable Hugging Face model. Do not confuse a resumable distributed checkpoint with the final Hugging Face artifact.
Confirm the expected model or adapter files, trainer state, metrics, and logs before evaluation or export.
## Evaluate
Use an independent dataset or a validation split fixed before training. Register it as a separate LlamaFactory dataset, retain stable sample IDs, and check that neither the prompts nor their reference answers occur in the training set. A smoke dataset or examples seen during training can validate plumbing but cannot establish a quality gain.
Install the repository's generation-metric dependencies with `python -m pip install -r requirements/metrics.txt` before using `predict_with_generate`; a training-only environment may not contain `jieba`, `nltk`, or `rouge-chinese`. Re-run `python -m pip check` after installation.
For instruction-following tasks, run generation on the held-out set with `do_predict: true` and `predict_with_generate: true`. Use the configuration for the artifact type instead of adapting QLoRA fields implicitly.
### QLoRA base and tuned runs
Use this configuration for both runs. Keep `model_name_or_path` on the same base model and add `adapter_name_or_path` only to the tuned run. Preserve the training run's quantization loading fields in both evaluation configurations; in particular, Ascend QLoRA requires `double_quantization: false`:
```yaml
model_name_or_path: <base-model-path>
# adapter_name_or_path: <adapter-path> # Tuned run only.
quantization_bit: 4
quantization_method: bnb
double_quantization: <same-as-training> # false for the repository's Ascend QLoRA path.
trust_remote_code: true
stage: sft
do_predict: true
finetuning_type: lora
eval_dataset: <held-out-dataset-name>
template: qwen3_8
# enable_thinking: false # Optional; set deliberately and keep identical in both runs.
cutoff_len: <evaluation-cutoff>
overwrite_cache: true
preprocessing_num_workers: <worker-count>
dataloader_num_workers: <worker-count>
output_dir: <unique-evaluation-output>
overwrite_output_dir: true
report_to: none
per_device_eval_batch_size: 1
predict_with_generate: true
do_sample: false
max_new_tokens: <generation-limit>
seed: <fixed-seed>
```
### HyperParallel full-SFT base and tuned runs
Use a full-tuning evaluation configuration with no adapter fields. Point the base run at the original model and the tuned run at the final Hugging Face artifact:
```yaml
model_name_or_path: <base-model-path-or-tuned-full-model-path>
trust_remote_code: true
stage: sft
do_predict: true
finetuning_type: full
eval_dataset: <held-out-dataset-name>
template: qwen3_8
# enable_thinking: false # Optional; set deliberately and keep identical in both runs.
cutoff_len: <evaluation-cutoff>
overwrite_cache: true
preprocessing_num_workers: <worker-count>
dataloader_num_workers: <worker-count>
output_dir: <unique-evaluation-output>
overwrite_output_dir: true
report_to: none
per_device_eval_batch_size: 1
predict_with_generate: true
do_sample: false
max_new_tokens: <generation-limit>
seed: <fixed-seed>
```
Do not carry `adapter_name_or_path`, `finetuning_type: lora`, or QLoRA quantization fields into the full-SFT configuration by accident. Prefer the deployment precision when it fits the evaluation hardware. If both full-model runs require on-the-fly 4-bit loading, add identical quantization fields to both configurations and disclose that their absolute scores may differ from unquantized evaluation.
Run each configuration with `llamafactory-cli train <eval.yaml>`. Despite the command name, `do_predict: true` performs prediction without training and writes `generated_predictions.jsonl` under `output_dir`.
Create one run for the base model and one for the tuned artifact:
- Give every run a separate output directory. Keep the held-out data, template, thinking settings, cutoff, decoding parameters, seed, and hardware policy identical.
Choose metrics before inspecting the outputs. Use deterministic checks for structured tasks, such as exact match, schema validity, required-field recall, tool-call validity, or task-specific unit tests. For open-ended answers, use a fixed rubric and blinded human or judge scoring, and retain representative wins, ties, and regressions. Token-level loss or perplexity can supplement generation metrics but should not replace task-level evaluation.
Report paired base and tuned results, the number of evaluated samples, the evaluation-data version, metric definitions, and decoding settings. Claim improvement only when the predefined acceptance criteria pass without unacceptable regressions. A decreasing training loss alone is not evidence that tuning improved behavior.
## Export a QLoRA adapter
Create the export configuration outside the model and checkpoint directories:
```yaml
model_name_or_path: <base-model-path>
adapter_name_or_path: <adapter-checkpoint-path>
template: qwen3_8
finetuning_type: lora
export_dir: <merged-model-output-path>
export_size: 5
export_device: cpu
export_legacy_format: false
```
Run `llamafactory-cli export <export.yaml>`, then reload the tokenizer, processor, and model from the merged directory. Retain the original adapter until the merged artifact passes this check. Full-SFT artifacts do not require adapter merging; reload their final Hugging Face directory directly.
## Verify direct inference and the local API
Perform direct inference first with an artifact-appropriate YAML. Then start the service locally:
```bash
API_HOST=127.0.0.1 API_PORT=8000 API_MODEL_NAME=<display-name> \
llamafactory-cli api <inference.yaml>
```
From another terminal, verify discovery and generation:
```bash
curl -sS http://127.0.0.1:8000/v1/models
curl -sS http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"<display-name>","messages":[{"role":"user","content":"<held-out-prompt>"}],"max_tokens":128}'
```
Inspect the response content, not only the HTTP status. If `API_KEY` is set, send a bearer token without printing or storing it. Keep the bind address local unless external exposure has been explicitly authorized and secured.
## Completion record
Report the code and package revisions, hardware, model and data versions, exact configs and commands, training and evaluation results, artifact paths, and API results. Include failures, fixes, and untested limitations. Remove credentials, internal addresses, usernames, absolute private paths, private model or dataset names, and sensitive samples before publishing.

62
.agents/skills/llamafactory-qwen38-sft/references/setup.md Normal file
View File

@@ -0,0 +1,62 @@
# Setup and compatibility
Run commands from the LlamaFactory repository root. Treat the current checkout's `README`, `pyproject.toml`, `requirements/`, examples, and source as authoritative.
## Record the starting state
Before installing or launching anything, record:
- the LlamaFactory commit and selected Python environment;
- accelerator type, driver and CUDA or CANN version, and selected device IDs;
- device utilization, process ownership, and available device memory;
- free space in the model-cache and training-output filesystems.
On Ascend, source the matching toolkit `set_env.sh` in every shell that invokes LlamaFactory. Confirm that the environment, `llamafactory-cli`, `torchrun`, and `accelerate` resolve from the intended installation.
## Install the selected stack
Use an isolated environment and install in this order:
1. Install the hardware-compatible PyTorch stack. On Ascend, use the torch and torch-npu pair pinned by `requirements/npu.txt` and ensure `pyyaml` is installed before importing torch-npu.
2. Install the LlamaFactory checkout with `python -m pip install -e .`.
3. Install only the optional dependencies required by the selected training path.
4. Install `transformers==5.8.0`, the version validated for this workflow and accepted by this checkout.
5. For HyperParallel, use the version declared compatible by the current LlamaFactory checkout. Until LlamaFactory declares a fixed version, use the latest `master`, resolve it to an immutable full commit SHA, and record that SHA with the run configuration. Before installing the source checkout, explicitly install its compatible build/runtime prerequisites with `python -m pip install numpy==1.26.4 pybind11`; the NumPy pin satisfies HyperParallel's declared `numpy>=1.26,<2.0` constraint instead of accepting the resolver's latest NumPy. Confirm that `HyperParallelArguments` and `parallelize_model` import, then use the smallest representative training smoke test as the compatibility gate. Install the source checkout with `python -m pip install --no-deps -e <hyperparallel-checkout>`. Do not select accelerator extras, because they can replace the torch stack selected in step 1.
For Ascend QLoRA, continue with [ascend-qlora.md](ascend-qlora.md). Do not load those build instructions for CUDA QLoRA or HyperParallel runs.
Do not disable LlamaFactory's version check. If `pyproject.toml` and `check_dependencies()` declare different bounds, obey the stricter bound. Record declared constraints and installed versions separately from the smoke-test result.
## Dependency gates
Run the gates required by the selected path:
```bash
python -m pip check
llamafactory-cli version
python -c 'import torch, transformers, accelerate, peft; assert transformers.__version__ == "5.8.0"; print(torch.__version__, transformers.__version__, accelerate.__version__, peft.__version__)'
python -c 'import torch_npu; print(torch_npu.__version__)' # Ascend only
python -c 'import bitsandbytes' # QLoRA only
python -c 'import hyper_parallel; from hyper_parallel.integration.llamafactory import HyperParallelArguments, parallelize_model; print(hyper_parallel.__file__, HyperParallelArguments())' # HyperParallel only
```
For Ascend QLoRA, complete the backend gate in [ascend-qlora.md](ascend-qlora.md); importing bitsandbytes alone does not prove that its NPU backend works. For HyperParallel, verify the printed checkout and both public integration symbols. When using `master`, an unrecorded moving branch or the development version string alone is not reproducible evidence. An import gate does not replace the training smoke test.
For this workflow, HyperParallel's declared `transformers>=5.11.0` requirement is a known metadata-only conflict with the validated `transformers==5.8.0` stack. Install HyperParallel from the recorded source revision with `--no-deps`, report this exact `pip check` finding, and do not treat this finding alone as a failed gate. Do not ignore any other dependency conflict. The HyperParallel integration imports and the real multi-device smoke test remain mandatory compatibility gates.
## Model compatibility and cache gates
Before a full download or training launch, confirm that the current checkout maps `Qwen3.8-27B` to `Qwen/Qwen3.8-27B` and template `qwen3_8`. Confirm with Transformers 5.8.0 that:
- `AutoConfig.model_type` is `qwen3_5`;
- the model class is `Qwen3_5ForConditionalGeneration`;
- the tokenizer, processor, and LlamaFactory template load.
Before declaring a local model complete, require:
- every shard named by `model.safetensors.index.json`, validated against the resolved upstream revision;
- `config.json`, tokenizer files, `preprocessor_config.json`, and `video_preprocessor_config.json`;
- no relevant incomplete download files;
- read access for the user that will run training.
Stop at a failed gate, preserve its log, and report the failure without substituting another model, backend, or training method.

54
.agents/skills/llamafactory-qwen38-sft/references/training.md Normal file
View File

@@ -0,0 +1,54 @@
# Training workflows
Use this reference after the applicable gates in [setup.md](setup.md) pass. Resolve dataset, sequence length, sample limit, gradient accumulation, duration, and output paths for the actual hardware instead of copying them from an example.
Quote YAML values that YAML 1.1 may coerce to booleans when the downstream option expects a string. In particular, disable checkpoint saves with `save_strategy: "no"`, not an unquoted `save_strategy: no`; PyYAML otherwise resolves `no` to `False`, which Transformers rejects as an invalid `SaveStrategy`.
## Path A: single-device 4-bit QLoRA
On Ascend, start from `examples/train_qlora/qwen3_lora_sft_bnb_npu.yaml`. On CUDA, start from `examples/train_qlora/qwen3_lora_sft_otfq.yaml`. Copy the selected file into the run directory, change the model to `Qwen/Qwen3.8-27B` and the template to `qwen3_8`, then resolve the dataset, output path, and run-size controls for the requested workflow.
Launch with one previously selected device:
```bash
ASCEND_RT_VISIBLE_DEVICES=<id> llamafactory-cli train <single-device-smoke.yaml>
```
Use `CUDA_VISIBLE_DEVICES` on CUDA. Accept the path only after measuring its actual loading and training peak. The output is a LoRA adapter and must later be loaded with the same base model.
## Path B: multi-device HyperParallel FSDP2 full SFT
### Ascend smoke-tested path
On Ascend, use `examples/ascend/qwen3_8_full_sft_hyper_fsdp2.yaml` directly as the repository training configuration. It already selects full SFT, HyperParallel, `qwen3_8`, batch size 1, and bf16; do not rebuild those settings from the Qwen3.5 example.
Copy `examples/accelerate/fsdp2_config_qwen38.yaml` into the run directory as well and set `num_processes` to the selected visible-device count. The example defaults to 16 processes; a smaller world size may require additional memory controls for full tuning. Its wrap classes use Qwen3.5 names because Qwen3.8 uses the `Qwen3_5` Transformers implementation.
Launch on the selected devices with the platform visibility variable:
```bash
ASCEND_RT_VISIBLE_DEVICES=<id0,id1,...> accelerate launch \
--config_file <fsdp2-smoke.yaml> \
src/train.py <multi-device-smoke.yaml>
```
This path is full tuning and does not use quantization. Begin with the repository example unchanged except for the model, data, output, and hardware-dependent run controls. Tune parallelism or memory settings only after that baseline is understood.
Retain the repository example's `bf16: true` because compute precision materially changes memory and training behavior. Do not repeat stable LlamaFactory defaults merely for visibility; write fields when overriding a default or when the resolved configuration must be portable across framework revisions.
### CUDA validation path
The repository does not currently ship a CUDA-specific Qwen3.8 HyperParallel example, and this workflow has not server-validated that combination. Do not present the Ascend result as CUDA support merely by replacing `ASCEND_RT_VISIBLE_DEVICES` with `CUDA_VISIBLE_DEVICES`.
When CUDA HyperParallel validation is explicitly requested, copy the checked-in training and Accelerate YAMLs into a run-specific directory, change only the model, data, output, device-count, and other demonstrated hardware-dependent controls, and launch with `CUDA_VISIBLE_DEVICES`. Apply the same real-model acceptance gate below and record the CUDA, PyTorch, Transformers, HyperParallel, and LlamaFactory revisions. Mark the result pending until the run completes; only a passing run may be reported as server-validated.
## Performance disclosure
Most Qwen3.8 layers use linear attention. Record whether `flash-linear-attention` and `causal-conv1d` enable the optimized path or Transformers falls back to its torch implementation. Do not compare or promise optimized throughput for a fallback run.
## Training acceptance
Before a formal run, execute a bounded smoke test with the selected training path and the real model. Confirm the intended backend and ranks, model loading, data preprocessing, forward, backward, and at least two consecutive optimizer updates with finite losses. A single step can finish before optimizer-state memory reaches its steady level.
For a requested training artifact, require a clean exit, complete expected files, and successful reload before evaluation, export, or serving. Record the environment, device count, exact command and configuration, duration, peak memory, loss or evaluation metrics, artifact paths, and reload result. Choose checkpoint retention deliberately, verify output space before training, and use a fresh output directory after the smoke test.

49
.claude/skills/llamafactory-qwen38-sft/SKILL.md Normal file
View File

@@ -0,0 +1,49 @@
---
name: llamafactory-qwen38-sft
description: Run end-to-end LlamaFactory SFT for Qwen/Qwen3.8-27B using single-device 4-bit QLoRA or multi-device HyperParallel FSDP2 full tuning, including setup, training validation, artifact evaluation, export, and local serving. Use only for Qwen3.8-27B.
---
# Qwen3.8-27B SFT with LlamaFactory
Drive LlamaFactory directly from environment setup through a verified local artifact. Do not build a separate agent runtime or training harness.
## Model contract
- Use the exact model `Qwen/Qwen3.8-27B`; do not substitute Qwen3-8B or Qwen3.5-27B.
- Use the LlamaFactory template `qwen3_8`. Transformers correctly exposes the architecture as `qwen3_5` / `Qwen3_5ForConditionalGeneration`.
- Treat the model as multimodal and choose text-only or multimodal data deliberately. Preserve the repository example's model-component defaults unless the requested tuning scope requires an override.
- Keep the `qwen3_8` template when changing thinking behavior. Record explicit `enable_thinking` or `reasoning_effort` overrides with the run configuration.
- Use single-device 4-bit QLoRA or multi-device HyperParallel with Accelerate FSDP2 full tuning. The checked-in HyperParallel recipe is smoke-tested on Ascend; treat CUDA HyperParallel and artifact-level completion as pending until they independently pass the applicable acceptance gates. Do not silently replace either path when a dependency or capacity gate fails.
## References
Load only the material needed for the requested stage:
- For environment creation, dependency gates, hardware selection, or model-cache validation, read [setup.md](references/setup.md).
- For the Ascend 4-bit QLoRA dependency path, additionally read [ascend-qlora.md](references/ascend-qlora.md).
- For training configuration, QLoRA, HyperParallel, or the training smoke test, read [training.md](references/training.md).
- For evaluation, artifact validation, export, inference, or local API verification, read [serving.md](references/serving.md).
- For the complete workflow, read them in that order.
## Workflow
1. Establish the requested scope, dataset, accelerator, selected devices, and time and disk limits.
2. Record the LlamaFactory revision and validate the environment and local model before training.
3. Validate and register custom data using the current repository documentation; preserve source data and keep held-out samples separate.
4. Resolve the training parameters and keep generated configuration, logs, and artifacts in explicit run-specific directories.
5. Run the smallest representative smoke test at the requested validation scope, then use a fresh output directory for a formal run.
6. When an artifact is requested, reload it before export or serving. Compare the base and tuned models with the same held-out prompts and decoding settings.
7. Report versions, hardware, configs, commands, metrics, artifact paths, failures, fixes, and untested limitations. Mark conclusions as code-confirmed, server-validated, or pending.
## Execution boundaries
- Pin the selected devices with the platform visibility variable and do not share occupied devices without explicit agreement.
- Run long downloads and training jobs with durable logs. Verify the real worker process and output artifacts; a detached launcher exiting successfully is not completion.
- Keep base weights, generated configs and logs, and training artifacts in separate paths. Record their resolved absolute paths.
- Bind services to `127.0.0.1` by default. External exposure requires explicit authorization, authentication, and a network plan.
- Never print or persist credentials.
- Do not alter model code merely to make the workflow pass. Preserve reproducible framework failures and treat their fixes as separately testable changes.
## Package maintenance
Treat `.agents/skills/llamafactory-qwen38-sft` as the canonical package. When changing it, update the `.claude/skills/llamafactory-qwen38-sft` compatibility mirror byte-for-byte and verify both directories with `diff -qr` before committing. This rule is for skill maintenance and does not add a step to an SFT run.

15
.claude/skills/llamafactory-qwen38-sft/references/ascend-qlora.md Normal file
View File

@@ -0,0 +1,15 @@
# Ascend 4-bit QLoRA dependency path
Read this reference only for single-device 4-bit QLoRA on Ascend. The LlamaFactory checkout's current README and `examples/train_qlora/qwen3_lora_sft_bnb_npu.yaml` remain authoritative.
Build the NPU backend from the `multi-backend-refactor` branch of `bitsandbytes-foundation/bitsandbytes` with CMake `COMPUTE_BACKEND=npu`; a regular CUDA-oriented bitsandbytes wheel is not an Ascend acceptance result.
Prefer building a wheel in an isolated build environment and installing that wheel into the training environment with `--no-deps`. Record the bitsandbytes commit and wheel checksum so the environment can be reproduced without letting the build toolchain replace the selected Python or PyTorch stack.
Before training:
1. Import bitsandbytes from the selected training environment and confirm that the NPU shared library, rather than a CUDA fallback, is loaded.
2. Run a small NPU tensor-quantization operation. Import success alone is insufficient.
3. Preserve `double_quantization: false` from the repository's NPU example unless a separately validated backend revision supports otherwise.
If import or the backend operation fails, preserve the build and runtime logs before troubleshooting. Inspect `libbitsandbytes_npu.so` with `readelf` and `ldd`, confirm that its dependencies resolve, and check whether its RPATH or RUNPATH refers to a removed build environment. Stop without replacing the requested QLoRA path with another tuning method when the backend gate remains unresolved.

137
.claude/skills/llamafactory-qwen38-sft/references/serving.md Normal file
View File

@@ -0,0 +1,137 @@
# Artifact evaluation and serving
Use this reference only after a training artifact exists and passes the reload requirement in [training.md](training.md).
## Identify the artifact
- A QLoRA run produces an adapter. Keep the original base model and load it with `adapter_name_or_path`.
- A HyperParallel full-SFT run produces a directly loadable Hugging Face model. Do not confuse a resumable distributed checkpoint with the final Hugging Face artifact.
Confirm the expected model or adapter files, trainer state, metrics, and logs before evaluation or export.
## Evaluate
Use an independent dataset or a validation split fixed before training. Register it as a separate LlamaFactory dataset, retain stable sample IDs, and check that neither the prompts nor their reference answers occur in the training set. A smoke dataset or examples seen during training can validate plumbing but cannot establish a quality gain.
Install the repository's generation-metric dependencies with `python -m pip install -r requirements/metrics.txt` before using `predict_with_generate`; a training-only environment may not contain `jieba`, `nltk`, or `rouge-chinese`. Re-run `python -m pip check` after installation.
For instruction-following tasks, run generation on the held-out set with `do_predict: true` and `predict_with_generate: true`. Use the configuration for the artifact type instead of adapting QLoRA fields implicitly.
### QLoRA base and tuned runs
Use this configuration for both runs. Keep `model_name_or_path` on the same base model and add `adapter_name_or_path` only to the tuned run. Preserve the training run's quantization loading fields in both evaluation configurations; in particular, Ascend QLoRA requires `double_quantization: false`:
```yaml
model_name_or_path: <base-model-path>
# adapter_name_or_path: <adapter-path> # Tuned run only.
quantization_bit: 4
quantization_method: bnb
double_quantization: <same-as-training> # false for the repository's Ascend QLoRA path.
trust_remote_code: true
stage: sft
do_predict: true
finetuning_type: lora
eval_dataset: <held-out-dataset-name>
template: qwen3_8
# enable_thinking: false # Optional; set deliberately and keep identical in both runs.
cutoff_len: <evaluation-cutoff>
overwrite_cache: true
preprocessing_num_workers: <worker-count>
dataloader_num_workers: <worker-count>
output_dir: <unique-evaluation-output>
overwrite_output_dir: true
report_to: none
per_device_eval_batch_size: 1
predict_with_generate: true
do_sample: false
max_new_tokens: <generation-limit>
seed: <fixed-seed>
```
### HyperParallel full-SFT base and tuned runs
Use a full-tuning evaluation configuration with no adapter fields. Point the base run at the original model and the tuned run at the final Hugging Face artifact:
```yaml
model_name_or_path: <base-model-path-or-tuned-full-model-path>
trust_remote_code: true
stage: sft
do_predict: true
finetuning_type: full
eval_dataset: <held-out-dataset-name>
template: qwen3_8
# enable_thinking: false # Optional; set deliberately and keep identical in both runs.
cutoff_len: <evaluation-cutoff>
overwrite_cache: true
preprocessing_num_workers: <worker-count>
dataloader_num_workers: <worker-count>
output_dir: <unique-evaluation-output>
overwrite_output_dir: true
report_to: none
per_device_eval_batch_size: 1
predict_with_generate: true
do_sample: false
max_new_tokens: <generation-limit>
seed: <fixed-seed>
```
Do not carry `adapter_name_or_path`, `finetuning_type: lora`, or QLoRA quantization fields into the full-SFT configuration by accident. Prefer the deployment precision when it fits the evaluation hardware. If both full-model runs require on-the-fly 4-bit loading, add identical quantization fields to both configurations and disclose that their absolute scores may differ from unquantized evaluation.
Run each configuration with `llamafactory-cli train <eval.yaml>`. Despite the command name, `do_predict: true` performs prediction without training and writes `generated_predictions.jsonl` under `output_dir`.
Create one run for the base model and one for the tuned artifact:
- Give every run a separate output directory. Keep the held-out data, template, thinking settings, cutoff, decoding parameters, seed, and hardware policy identical.
Choose metrics before inspecting the outputs. Use deterministic checks for structured tasks, such as exact match, schema validity, required-field recall, tool-call validity, or task-specific unit tests. For open-ended answers, use a fixed rubric and blinded human or judge scoring, and retain representative wins, ties, and regressions. Token-level loss or perplexity can supplement generation metrics but should not replace task-level evaluation.
Report paired base and tuned results, the number of evaluated samples, the evaluation-data version, metric definitions, and decoding settings. Claim improvement only when the predefined acceptance criteria pass without unacceptable regressions. A decreasing training loss alone is not evidence that tuning improved behavior.
## Export a QLoRA adapter
Create the export configuration outside the model and checkpoint directories:
```yaml
model_name_or_path: <base-model-path>
adapter_name_or_path: <adapter-checkpoint-path>
template: qwen3_8
finetuning_type: lora
export_dir: <merged-model-output-path>
export_size: 5
export_device: cpu
export_legacy_format: false
```
Run `llamafactory-cli export <export.yaml>`, then reload the tokenizer, processor, and model from the merged directory. Retain the original adapter until the merged artifact passes this check. Full-SFT artifacts do not require adapter merging; reload their final Hugging Face directory directly.
## Verify direct inference and the local API
Perform direct inference first with an artifact-appropriate YAML. Then start the service locally:
```bash
API_HOST=127.0.0.1 API_PORT=8000 API_MODEL_NAME=<display-name> \
llamafactory-cli api <inference.yaml>
```
From another terminal, verify discovery and generation:
```bash
curl -sS http://127.0.0.1:8000/v1/models
curl -sS http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"<display-name>","messages":[{"role":"user","content":"<held-out-prompt>"}],"max_tokens":128}'
```
Inspect the response content, not only the HTTP status. If `API_KEY` is set, send a bearer token without printing or storing it. Keep the bind address local unless external exposure has been explicitly authorized and secured.
## Completion record
Report the code and package revisions, hardware, model and data versions, exact configs and commands, training and evaluation results, artifact paths, and API results. Include failures, fixes, and untested limitations. Remove credentials, internal addresses, usernames, absolute private paths, private model or dataset names, and sensitive samples before publishing.

62
.claude/skills/llamafactory-qwen38-sft/references/setup.md Normal file
View File

@@ -0,0 +1,62 @@
# Setup and compatibility
Run commands from the LlamaFactory repository root. Treat the current checkout's `README`, `pyproject.toml`, `requirements/`, examples, and source as authoritative.
## Record the starting state
Before installing or launching anything, record:
- the LlamaFactory commit and selected Python environment;
- accelerator type, driver and CUDA or CANN version, and selected device IDs;
- device utilization, process ownership, and available device memory;
- free space in the model-cache and training-output filesystems.
On Ascend, source the matching toolkit `set_env.sh` in every shell that invokes LlamaFactory. Confirm that the environment, `llamafactory-cli`, `torchrun`, and `accelerate` resolve from the intended installation.
## Install the selected stack
Use an isolated environment and install in this order:
1. Install the hardware-compatible PyTorch stack. On Ascend, use the torch and torch-npu pair pinned by `requirements/npu.txt` and ensure `pyyaml` is installed before importing torch-npu.
2. Install the LlamaFactory checkout with `python -m pip install -e .`.
3. Install only the optional dependencies required by the selected training path.
4. Install `transformers==5.8.0`, the version validated for this workflow and accepted by this checkout.
5. For HyperParallel, use the version declared compatible by the current LlamaFactory checkout. Until LlamaFactory declares a fixed version, use the latest `master`, resolve it to an immutable full commit SHA, and record that SHA with the run configuration. Before installing the source checkout, explicitly install its compatible build/runtime prerequisites with `python -m pip install numpy==1.26.4 pybind11`; the NumPy pin satisfies HyperParallel's declared `numpy>=1.26,<2.0` constraint instead of accepting the resolver's latest NumPy. Confirm that `HyperParallelArguments` and `parallelize_model` import, then use the smallest representative training smoke test as the compatibility gate. Install the source checkout with `python -m pip install --no-deps -e <hyperparallel-checkout>`. Do not select accelerator extras, because they can replace the torch stack selected in step 1.
For Ascend QLoRA, continue with [ascend-qlora.md](ascend-qlora.md). Do not load those build instructions for CUDA QLoRA or HyperParallel runs.
Do not disable LlamaFactory's version check. If `pyproject.toml` and `check_dependencies()` declare different bounds, obey the stricter bound. Record declared constraints and installed versions separately from the smoke-test result.
## Dependency gates
Run the gates required by the selected path:
```bash
python -m pip check
llamafactory-cli version
python -c 'import torch, transformers, accelerate, peft; assert transformers.__version__ == "5.8.0"; print(torch.__version__, transformers.__version__, accelerate.__version__, peft.__version__)'
python -c 'import torch_npu; print(torch_npu.__version__)' # Ascend only
python -c 'import bitsandbytes' # QLoRA only
python -c 'import hyper_parallel; from hyper_parallel.integration.llamafactory import HyperParallelArguments, parallelize_model; print(hyper_parallel.__file__, HyperParallelArguments())' # HyperParallel only
```
For Ascend QLoRA, complete the backend gate in [ascend-qlora.md](ascend-qlora.md); importing bitsandbytes alone does not prove that its NPU backend works. For HyperParallel, verify the printed checkout and both public integration symbols. When using `master`, an unrecorded moving branch or the development version string alone is not reproducible evidence. An import gate does not replace the training smoke test.
For this workflow, HyperParallel's declared `transformers>=5.11.0` requirement is a known metadata-only conflict with the validated `transformers==5.8.0` stack. Install HyperParallel from the recorded source revision with `--no-deps`, report this exact `pip check` finding, and do not treat this finding alone as a failed gate. Do not ignore any other dependency conflict. The HyperParallel integration imports and the real multi-device smoke test remain mandatory compatibility gates.
## Model compatibility and cache gates
Before a full download or training launch, confirm that the current checkout maps `Qwen3.8-27B` to `Qwen/Qwen3.8-27B` and template `qwen3_8`. Confirm with Transformers 5.8.0 that:
- `AutoConfig.model_type` is `qwen3_5`;
- the model class is `Qwen3_5ForConditionalGeneration`;
- the tokenizer, processor, and LlamaFactory template load.
Before declaring a local model complete, require:
- every shard named by `model.safetensors.index.json`, validated against the resolved upstream revision;
- `config.json`, tokenizer files, `preprocessor_config.json`, and `video_preprocessor_config.json`;
- no relevant incomplete download files;
- read access for the user that will run training.
Stop at a failed gate, preserve its log, and report the failure without substituting another model, backend, or training method.

54
.claude/skills/llamafactory-qwen38-sft/references/training.md Normal file
View File

@@ -0,0 +1,54 @@
# Training workflows
Use this reference after the applicable gates in [setup.md](setup.md) pass. Resolve dataset, sequence length, sample limit, gradient accumulation, duration, and output paths for the actual hardware instead of copying them from an example.
Quote YAML values that YAML 1.1 may coerce to booleans when the downstream option expects a string. In particular, disable checkpoint saves with `save_strategy: "no"`, not an unquoted `save_strategy: no`; PyYAML otherwise resolves `no` to `False`, which Transformers rejects as an invalid `SaveStrategy`.
## Path A: single-device 4-bit QLoRA
On Ascend, start from `examples/train_qlora/qwen3_lora_sft_bnb_npu.yaml`. On CUDA, start from `examples/train_qlora/qwen3_lora_sft_otfq.yaml`. Copy the selected file into the run directory, change the model to `Qwen/Qwen3.8-27B` and the template to `qwen3_8`, then resolve the dataset, output path, and run-size controls for the requested workflow.
Launch with one previously selected device:
```bash
ASCEND_RT_VISIBLE_DEVICES=<id> llamafactory-cli train <single-device-smoke.yaml>
```
Use `CUDA_VISIBLE_DEVICES` on CUDA. Accept the path only after measuring its actual loading and training peak. The output is a LoRA adapter and must later be loaded with the same base model.
## Path B: multi-device HyperParallel FSDP2 full SFT
### Ascend smoke-tested path
On Ascend, use `examples/ascend/qwen3_8_full_sft_hyper_fsdp2.yaml` directly as the repository training configuration. It already selects full SFT, HyperParallel, `qwen3_8`, batch size 1, and bf16; do not rebuild those settings from the Qwen3.5 example.
Copy `examples/accelerate/fsdp2_config_qwen38.yaml` into the run directory as well and set `num_processes` to the selected visible-device count. The example defaults to 16 processes; a smaller world size may require additional memory controls for full tuning. Its wrap classes use Qwen3.5 names because Qwen3.8 uses the `Qwen3_5` Transformers implementation.
Launch on the selected devices with the platform visibility variable:
```bash
ASCEND_RT_VISIBLE_DEVICES=<id0,id1,...> accelerate launch \
--config_file <fsdp2-smoke.yaml> \
src/train.py <multi-device-smoke.yaml>
```
This path is full tuning and does not use quantization. Begin with the repository example unchanged except for the model, data, output, and hardware-dependent run controls. Tune parallelism or memory settings only after that baseline is understood.
Retain the repository example's `bf16: true` because compute precision materially changes memory and training behavior. Do not repeat stable LlamaFactory defaults merely for visibility; write fields when overriding a default or when the resolved configuration must be portable across framework revisions.
### CUDA validation path
The repository does not currently ship a CUDA-specific Qwen3.8 HyperParallel example, and this workflow has not server-validated that combination. Do not present the Ascend result as CUDA support merely by replacing `ASCEND_RT_VISIBLE_DEVICES` with `CUDA_VISIBLE_DEVICES`.
When CUDA HyperParallel validation is explicitly requested, copy the checked-in training and Accelerate YAMLs into a run-specific directory, change only the model, data, output, device-count, and other demonstrated hardware-dependent controls, and launch with `CUDA_VISIBLE_DEVICES`. Apply the same real-model acceptance gate below and record the CUDA, PyTorch, Transformers, HyperParallel, and LlamaFactory revisions. Mark the result pending until the run completes; only a passing run may be reported as server-validated.
## Performance disclosure
Most Qwen3.8 layers use linear attention. Record whether `flash-linear-attention` and `causal-conv1d` enable the optimized path or Transformers falls back to its torch implementation. Do not compare or promise optimized throughput for a fallback run.
## Training acceptance
Before a formal run, execute a bounded smoke test with the selected training path and the real model. Confirm the intended backend and ranks, model loading, data preprocessing, forward, backward, and at least two consecutive optimizer updates with finite losses. A single step can finish before optimizer-state memory reaches its steady level.
For a requested training artifact, require a clean exit, complete expected files, and successful reload before evaluation, export, or serving. Record the environment, device count, exact command and configuration, duration, peak memory, loss or evaluation metrics, artifact paths, and reload result. Choose checkpoint retention deliberately, verify output space before training, and use a fresh output directory after the smoke test.

20
examples/accelerate/fsdp2_config_qwen38.yaml Normal file
View File

@@ -0,0 +1,20 @@
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
fsdp_version: 2
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_transformer_layer_cls_to_wrap: Qwen3_5DecoderLayer,Qwen3_5VisionBlock
fsdp_cpu_ram_efficient_loading: true
fsdp_offload_params: false
fsdp_reshard_after_forward: true
fsdp_state_dict_type: FULL_STATE_DICT
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 16 # Change together with the visible-device list for your hardware.
rdzv_backend: static
same_network: true
use_cpu: false

48
examples/ascend/qwen3_8_full_sft_hyper_fsdp2.yaml Normal file
View File

@@ -0,0 +1,48 @@
# Start HyperParallel FSDP2 full fine-tuning on Ascend NPU
# Usage:
# ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 accelerate launch \
# --config_file examples/accelerate/fsdp2_config_qwen38.yaml \
# src/train.py examples/ascend/qwen3_8_full_sft_hyper_fsdp2.yaml
#
# Note: Change both the visible-device list and `num_processes` for your hardware.
### model
model_name_or_path: Qwen/Qwen3.8-27B
trust_remote_code: true
use_v1_kernels: true
flash_attn: fa2
### method
stage: sft
do_train: true
finetuning_type: full
use_hyper_parallel: true
### dataset
dataset: alpaca_en_demo
template: qwen3_8
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4
### output
output_dir: saves/Qwen3.8-27B/full/sft
logging_steps: 1
save_steps: 500
max_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow]
### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 1800
resume_from_checkpoint: null