423A35C7/LLaMA-Factory
1
0
Fork 0
mirror of https://github.com/hiyouga/LLaMA-Factory.git synced 2025-10-16 00:28:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #6152 from hiyouga/hiyouga/add_num_proc_in_data_load

Browse Source 
[data] add num_proc in load_dataset

Former-commit-id: d8258ba7e792d5f17ae80d5e8b303e8fa820f162
This commit is contained in:
hoshi-hiyouga 2024-11-27 00:16:15 +08:00 committed by GitHub
commit fcf1a3df62
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
src/llamafactory/data/loader.py
View File

@ -128,6 +128,7 @@ def _load_single_dataset(
cache_dir=model_args.cache_dir, cache_dir=model_args.cache_dir,
token=model_args.hf_hub_token, token=model_args.hf_hub_token,
streaming=data_args.streaming, streaming=data_args.streaming,
num_proc=data_args.preprocessing_num_workers,
trust_remote_code=True, trust_remote_code=True,
) )