101 Commits

Author	SHA1	Message	Date
hiyouga	f6b2bcfa16	fix #2420 ... Former-commit-id: 19d33ede13	2024-02-04 15:51:47 +08:00
hiyouga	b1064d2f9b	bump up transformers version ... Former-commit-id: 38e63bfd28	2024-02-04 00:01:16 +08:00
hiyouga	0fc8612b97	add hint for freeze #2412 ... Former-commit-id: 6545c02790	2024-02-03 23:38:56 +08:00
hiyouga	a9e58740f5	fix #2376 ... Former-commit-id: 4ecadc3512	2024-02-03 23:14:31 +08:00
hiyouga	7beeae2209	fix autoset attn impl, update data readme ... Former-commit-id: 521ad76552	2024-01-31 11:58:07 +08:00
hiyouga	b8a827faeb	fix #2320 ... Former-commit-id: 2bc30763e9	2024-01-24 16:19:18 +08:00
ldwang	323ec3f89f	Add patch_mixtral_replace_moe_impl for full training Mitral using DeepSpeed Zero3. ... Signed-off-by: ldwang <ftgreat@gmail.com> Former-commit-id: c284665425	2024-01-24 15:25:31 +08:00
ldwang	db500a2bb6	Add patch_mixtral_replace_moe_impl for full training Mitral using DeepSpeed Zero3. ... Signed-off-by: ldwang <ftgreat@gmail.com> Former-commit-id: 18923b1402	2024-01-24 14:43:16 +08:00
hiyouga	60a042cc16	add hint ... Former-commit-id: e4ba1deedf	2024-01-22 23:32:01 +08:00
hoshi-hiyouga	68977c8ca4	Update patcher.py ... Former-commit-id: bdc9eff635	2024-01-22 23:27:39 +08:00
A-Cepheus	f00ad6b4f8	🐞 fix: typo ... Former-commit-id: b06a31e76a	2024-01-22 16:04:39 +08:00
A-Cepheus	39d9aba166	🐞 fix: typo, move MoE fix to patcher ... Former-commit-id: 319a72b48d	2024-01-22 16:01:58 +08:00
A-Cepheus	8985c43033	fix: ZeRO3 does not work with MoE models ... Former-commit-id: e1d5c98519	2024-01-22 15:21:14 +08:00
hiyouga	fb2d563be5	fix #2268 ... Former-commit-id: e0a717aa3a	2024-01-21 14:11:38 +08:00
hiyouga	b27e91222c	format style ... Former-commit-id: 638234ceee	2024-01-20 20:15:56 +08:00
hiyouga	69e8925249	support longlora for main branch ... Former-commit-id: 38af076a75	2024-01-20 19:25:22 +08:00
hoshi-hiyouga	b36dea7e7b	Merge pull request #2201 from liu-zichen/token_embed_resize ... support resize embed for zero3 Former-commit-id: bb92cdd0db	2024-01-20 17:45:38 +08:00
hiyouga	edf725208d	add upcast_lmhead option ... Former-commit-id: 8cbe4e9609	2024-01-19 23:54:25 +08:00
hiyouga	ffde3d94bf	set use_reentrant=False ... Former-commit-id: 0ff9a1fb4f	2024-01-19 23:29:54 +08:00
hiyouga	e199967391	add bf16 lora option ... Former-commit-id: b6ec112beb	2024-01-19 16:29:03 +08:00
hiyouga	4e3bfb799d	support function calling ... Former-commit-id: d9f1cae351	2024-01-18 09:54:23 +08:00
liuzc	61bc5bd0dd	support resize embed for zero3 ... Former-commit-id: a5f6a7f4fb	2024-01-16 15:16:20 +08:00
hiyouga	61960189b2	fix #1789 ... Former-commit-id: 4571068e1e	2024-01-09 18:31:27 +08:00
hiyouga	f53bc7d9a0	fix #2127 ... Former-commit-id: ebee4f6a2a	2024-01-09 14:49:13 +08:00
hiyouga	6bbcf5ad16	improve model export ... Former-commit-id: d2a676c8ba	2024-01-05 18:51:49 +08:00
hiyouga	7c50959fe6	fix #2098 ... Former-commit-id: f6fdd83f8a	2024-01-05 17:11:26 +08:00
hiyouga	dc8714a003	fix #2081 ... Former-commit-id: 33f2c0d4f8	2024-01-04 23:19:08 +08:00
hiyouga	3fff67e3c7	fix dispatch ... Former-commit-id: 1696698eb9	2024-01-03 16:33:16 +08:00
hiyouga	eb021ca748	fix valuehead patch ... Former-commit-id: 24d8d6f224	2024-01-03 16:19:23 +08:00
hiyouga	1a86cc3078	fix rm server ... Former-commit-id: 55021097d5	2024-01-03 15:30:46 +08:00
hiyouga	afa3c86391	fix version ... Former-commit-id: 47da742fc9	2023-12-29 04:53:36 +08:00
hiyouga	498b83d77a	fix bug ... Former-commit-id: e4bb846c43	2023-12-24 19:20:12 +08:00
hiyouga	2a3980d6ba	update loader ... Former-commit-id: 6629087e12	2023-12-24 19:10:23 +08:00
hiyouga	5d440f978e	update patcher ... Former-commit-id: e44b82ee24	2023-12-23 15:24:27 +08:00
hiyouga	a7fc20cb2d	fix unsloth dtype ... Former-commit-id: 779cfefb78	2023-12-23 01:59:49 +08:00
hiyouga	938c4cb132	fix dpo trainer ... Former-commit-id: 074745b170	2023-12-23 01:51:55 +08:00
hiyouga	f0d405f392	support unsloth ... Former-commit-id: 7aad0b889d	2023-12-23 00:14:33 +08:00
ShaneTian	3eaabe12aa	Fix slow model initialization in bfloat16 dtype. ... Former-commit-id: d032daa4bd	2023-12-22 16:27:28 +08:00
hiyouga	fee0fef052	fix ds zero3 check ... Former-commit-id: 083355fc05	2023-12-21 01:19:22 +08:00
hiyouga	de6be321b9	match version ... Former-commit-id: af0194e6d9	2023-12-20 22:17:35 +08:00
hiyouga	23a875a8b1	improve quantization ... Former-commit-id: 624cc21281	2023-12-20 18:27:16 +08:00
hiyouga	c1233ab65f	add max_memory for gptq #1923 ... Former-commit-id: c4a3977ad7	2023-12-20 18:15:17 +08:00
hiyouga	a862ce636f	fix mixtral inference #1821 ... Former-commit-id: f86857bd9e	2023-12-20 15:11:15 +08:00
hiyouga	e06b9c4fa1	fix #1900 ... Former-commit-id: 0c6ab7c75e	2023-12-19 17:21:46 +08:00
hiyouga	8154b4bdf6	fix #1742 ... Former-commit-id: 870426ff70	2023-12-16 20:50:45 +08:00
hiyouga	7c362509a6	add noisy mean initialization #1815 ... Former-commit-id: a66186b872	2023-12-16 19:47:51 +08:00
hiyouga	4e75ca1222	support dpo-ftx ... Former-commit-id: b87c74289d	2023-12-16 19:21:41 +08:00
hiyouga	f0f9d253d8	support autogptq in llama board #246 ... Former-commit-id: 71389be37c	2023-12-16 16:31:30 +08:00
hiyouga	7dbc670902	support quantization in export model ... Former-commit-id: 3524aa1e58	2023-12-15 23:44:50 +08:00
hiyouga	2db4cfab40	update dc link ... Former-commit-id: 87ef3f47b5	2023-12-15 22:11:31 +08:00