41 Commits

Author	SHA1	Message	Date
hiyouga	a9db89a025	update data readme (zh) ... Former-commit-id: cc8ffa10d8	2023-11-02 23:42:49 +08:00
hiyouga	a1b0655457	support sharegpt format, add datasets ... Former-commit-id: a837172413	2023-11-02 23:10:04 +08:00
hiyouga	22b3c913e9	fix #1325 ... Former-commit-id: 083787dbfe	2023-11-01 23:38:49 +08:00
hiyouga	fcfcac4858	support dataset cache ... Former-commit-id: 3fe7df628d	2023-10-26 21:48:45 +08:00
hiyouga	84e27a1c0b	remove filter in preprocess ... Former-commit-id: 2caf91f824	2023-10-23 23:46:02 +08:00
hiyouga	4930118761	fix #1218 ... Former-commit-id: 7a11a42dfd	2023-10-19 16:17:41 +08:00
hiyouga	e585c789ce	fix #1184 ... Former-commit-id: af18b0dce7	2023-10-14 19:20:11 +08:00
hiyouga	9ef9cb316b	fix webui ... Former-commit-id: b240b6792f	2023-10-13 16:27:59 +08:00
hiyouga	141937ead6	fix aquila template, repair sft packing mechanism ... Former-commit-id: be420e4179	2023-10-10 18:49:55 +08:00
hiyouga	f88088c43d	fix bug in packed sft dataset ... Former-commit-id: de19614306	2023-09-28 01:16:46 +08:00
hiyouga	8a8ba08bf7	tiny fix ... Former-commit-id: d2ebd225db	2023-09-28 01:02:11 +08:00
hiyouga	d2ce9b879b	fix bug in pretraining ... Former-commit-id: b3fbba57eb	2023-09-28 00:45:20 +08:00
hiyouga	108c31e1fc	support LongLoRA ... Former-commit-id: 90375f600d	2023-09-27 21:55:50 +08:00
hiyouga	4581d09fa6	fix #944 ... Former-commit-id: 338b8664ed	2023-09-21 19:51:02 +08:00
hiyouga	5377d0bf95	fix #896 ... Former-commit-id: 8632bff811	2023-09-14 18:37:34 +08:00
hiyouga	f865d0bd51	fix lora target ... Former-commit-id: a51b7c98ac	2023-09-09 17:04:45 +08:00
hiyouga	f74b980650	fix baichuan templates ... Former-commit-id: 85b1f6632a	2023-09-07 18:54:14 +08:00
hiyouga	3b5a9c60b6	fix #763 ... Former-commit-id: 370bdb6e43	2023-09-01 23:13:05 +08:00
hiyouga	a4fd976048	refactor dataset_attr, add eos in pt, fix #757 ... Former-commit-id: a9d1fb72f7	2023-09-01 19:00:45 +08:00
hiyouga	a46f277477	fix streaming in pt stage #548 #549 ... Former-commit-id: b0ed0dec5e	2023-08-17 17:59:26 +08:00
hiyouga	edc15c62fa	fix system prompt ... Former-commit-id: 7407d9daa1	2023-08-16 01:35:52 +08:00
hiyouga	79f4ba0d26	Release v0.1.6 ... Former-commit-id: a48cb0d474	2023-08-11 23:25:57 +08:00
hiyouga	21bf79e72b	add defaults ... Former-commit-id: d3844e97e3	2023-08-11 13:56:26 +08:00
hiyouga	f1485ab927	fix baichuan template ... Former-commit-id: 9c6dd10514	2023-08-11 13:45:47 +08:00
hiyouga	abdfa26d06	support DPO training (2305.18290) ... Former-commit-id: 3ec4351cfd	2023-08-11 03:02:53 +08:00
hiyouga	6404167ab7	support val set in streaming mode ... Former-commit-id: d86ea314a1	2023-08-09 23:00:26 +08:00
hiyouga	28a807472b	fix rm #420, fix template #426, fix #423 ... Former-commit-id: 39cd8b6989	2023-08-09 16:23:31 +08:00
hiyouga	77aa9853fb	fix tokenizer #417 ... Former-commit-id: eecc4b2131	2023-08-08 23:59:41 +08:00
hiyouga	9e49438c41	fix bug ... Former-commit-id: caa0eda27d	2023-08-08 21:28:28 +08:00
hiyouga	c796c542c8	fix chatml template #408 ... Former-commit-id: a9980617f5	2023-08-08 17:44:39 +08:00
hiyouga	733b395822	update readme ... Former-commit-id: 20cf27976f	2023-08-07 15:02:02 +08:00
hiyouga	65369ecf48	fix bos and eos token ... Former-commit-id: d87c8fd8ab	2023-08-04 23:55:57 +08:00
hiyouga	dbb284b5a2	fix encode ... Former-commit-id: 8172ad1b5e	2023-08-04 23:27:55 +08:00
hiyouga	ea045b0e5b	support chatml safe encoding ... Former-commit-id: b4852f9406	2023-08-04 23:14:28 +08:00
hiyouga	b32ed1d7be	support interleave probs ... Former-commit-id: 69744c17e8	2023-08-04 21:27:35 +08:00
hiyouga	de407b59ea	fix bug in preprocessing ... Former-commit-id: 968ce0dcce	2023-08-02 01:10:28 +08:00
hiyouga	43762f96e6	fix #296 ... Former-commit-id: e3f80774c4	2023-08-01 18:43:53 +08:00
hiyouga	e80b75b560	support streaming data, fix #284 #274 #268 ... Former-commit-id: 0411a4b3e1	2023-07-31 23:33:00 +08:00
hiyouga	f769c2d3fc	update web UI, support rm predict #210 ... Former-commit-id: ed0e186a13	2023-07-21 13:27:27 +08:00
hiyouga	64b4f71673	simplify code ... Former-commit-id: 67a2773074	2023-07-20 15:08:57 +08:00
hiyouga	a696148d6b	modity code structure ... Former-commit-id: f751376613	2023-07-15 16:54:28 +08:00