21 Commits

Author	SHA1	Message	Date
hoshi-hiyouga	e8e6af2651	Merge branch 'main' into main	2024-07-01 21:01:09 +08:00
hiyouga	1771251ce3	fix #4402 #4617 ... Deprecate reserved_label_len arg	2024-07-01 01:19:27 +08:00
hiyouga	38b6b0f52e	tiny fix	2024-06-16 01:06:41 +08:00
ancv	04315c3d92	remove some unused params	2024-06-15 23:00:55 +07:00
hiyouga	d87108daa6	add license	2024-06-15 17:54:33 +08:00
hiyouga	6baafd4eb3	fix #4221	2024-06-13 02:48:21 +08:00
ancv	b2c367bc61	implement efficient packing without cross-contamination attention	2024-06-12 11:56:01 +07:00
hoshi-hiyouga	0c29233237	Update pretrain.py	2024-06-11 17:02:14 +08:00
d	6979f3f848	经过大量的增量预训练，进行对比试验，发现这个bug：llama3在预训练时使用的tokenizer.eos_toke是'<|end_of_text|>' ，这里在每条数据后面也得用这个，而不是'<|eot_id|>'，否则很容易导致严重的性能下降	2024-06-11 16:23:40 +08:00
hiyouga	ccc8b64cc2	update data processors	2024-06-07 04:15:40 +08:00
hoshi-hiyouga	181dbb0d05	Merge pull request #4009 from AlongWY/main ... supervised packing with greedy knapsack algorithm	2024-06-07 03:48:46 +08:00
hoshi-hiyouga	c09ad8bab3	Update supervised.py	2024-06-07 03:42:08 +08:00
hoshi-hiyouga	788e8232fc	Update supervised.py	2024-06-07 03:38:23 +08:00
hoshi-hiyouga	8cecade708	Update supervised.py	2024-06-07 03:38:04 +08:00
hiyouga	74f96efef9	rename files	2024-06-07 00:09:06 +08:00
hiyouga	149610c636	fix ppo dataset bug #4012	2024-06-06 19:03:20 +08:00
ylfeng	b47e317447	remove empty line	2024-05-31 21:43:08 +08:00
ylfeng	84aee57901	fix eos	2024-05-31 21:40:41 +08:00
ylfeng	f9db439cb7	supervised packing with greedy knapsack algorithm	2024-05-31 15:33:54 +08:00
hiyouga	5581cb2e4e	update readme	2024-05-27 18:14:02 +08:00
hiyouga	3a023bca2a	refactor data preprocessing, fix mllm rlhf	2024-05-24 04:08:25 +08:00