4 Commits

Author	SHA1	Message	Date
hoshi-hiyouga	bf3de9bfe8	Update pretrain.py ... Former-commit-id: 0c292332374fb96c3fc753abde42d070a0c1dca8	2024-06-11 17:02:14 +08:00
d	da39715085	经过大量的增量预训练，进行对比试验，发现这个bug：llama3在预训练时使用的tokenizer.eos_toke是'<|end_of_text|>' ，这里在每条数据后面也得用这个，而不是'<|eot_id|>'，否则很容易导致严重的性能下降 ... Former-commit-id: 6979f3f8480755604d8aea8164f6418126e094c5	2024-06-11 16:23:40 +08:00
hiyouga	e0aadd4b34	fix ppo dataset bug #4012 ... Former-commit-id: 149610c636bbb974e546d13fa302884ea65a6d38	2024-06-06 19:03:20 +08:00
hiyouga	3e729798df	refactor data preprocessing, fix mllm rlhf ... Former-commit-id: 3a023bca2a502810a436cfba7708df164754ea62	2024-05-24 04:08:25 +08:00