423A35C7/LLaMA-Factory
1
0
Fork 0
mirror of https://github.com/hiyouga/LLaMA-Factory.git synced 2025-11-06 10:52:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix #2397

Browse Source 
Former-commit-id: 7404692808f2288d539668d364965ad104dacadb
This commit is contained in:
hiyouga 2024-02-03 23:45:31 +08:00
parent 80c0ad44a3
commit dcca5901a6
1 changed files with 0 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
src/llmtuner/data/loader.py
View File

@ -155,9 +155,6 @@ def get_dataset(
dataset = dataset.to_iterable_dataset() dataset = dataset.to_iterable_dataset()
return dataset return dataset
if data_args.streaming:
raise ValueError("Turn off dataset streaming to save cache files.")
with training_args.main_process_first(desc="load dataset"): with training_args.main_process_first(desc="load dataset"):
all_datasets = [] all_datasets = []
for dataset_attr in get_dataset_list(data_args): # TODO: add split for dataset_attr in get_dataset_list(data_args): # TODO: add split