67 Commits

Author SHA1 Message Date
hiyouga
654116c0b1 fix #4556
Former-commit-id: 59e0b4f616736ede37cc37a13346b547f5a2d4e7
2024-06-26 19:43:16 +08:00
hiyouga
d519c2fde5 tiny fix
Former-commit-id: 41086059b12ecb7827eb390294e315068ff9c2e6
2024-06-25 01:15:19 +08:00
hoshi-hiyouga
709bbc1d92 Merge pull request #4417 from mMrBun/main
Add tool_format parameter to rewrite templates for different function call formats.

Former-commit-id: def6d280db3a9fe468b05503bcd9929c83c6c19b
2024-06-24 23:17:55 +08:00
hoshi-hiyouga
b7f5cfde6e Update template.py
Former-commit-id: 1240bd57d8a21540c636a6da839e6b3112d1395a
2024-06-24 23:12:59 +08:00
hoshi-hiyouga
673f27a59e Update loader.py
Former-commit-id: dddfd516ee66e9937e21f05300832aab45034b12
2024-06-24 23:06:18 +08:00
hiyouga
47651a94a3 fix #4410
Former-commit-id: fca893d73c3d7bbb87a816522f2e1568d3e9c612
2024-06-24 22:34:31 +08:00
mMrBun
c0e005e2ea Add tool_format to overwrite tool formatter template
Former-commit-id: 20e2e6fdcb0cd1771906be035745a2d9fcd3e138
2024-06-22 02:13:23 +08:00
hiyouga
98abb5c900 remove dup template
Former-commit-id: db9a1912e3551394039cc57b4913f03e8f9aa29d
2024-06-22 01:31:32 +08:00
hiyouga
3d72b1a856 fix jinja template
Former-commit-id: 2b596fb55ff689d2e488d9a9bbab98f70f356c3c
2024-06-19 20:03:50 +08:00
hiyouga
7735456561 fix templates
Former-commit-id: 4cff6a4ad55b24bf57db6be5cf817180c1ea5626
2024-06-19 17:44:05 +08:00
hiyouga
c9557241f6 fix bug
Former-commit-id: 6d2bf216ac3a48450e861148ce664dad717fd019
2024-06-19 03:49:23 +08:00
hiyouga
e73a235a38 use prefix to replace force system
Former-commit-id: 4f22eae8f405de918237d406e5e9847592925565
2024-06-19 03:39:52 +08:00
hiyouga
bccc852f76 fix tool formatter, allow parallel function #4362
Former-commit-id: cd75b1fe9d91fb52a9ae6de7435302ff06b4d933
2024-06-19 03:23:51 +08:00
hoshi-hiyouga
6db02615d4 Merge pull request #4173 from mMrBun/main
Implemented the tool_formatter and tool_extractor for glm4 and Qwen2 tool_format

Former-commit-id: c0ca42566c6aeccd8d384377510690eafef10995
2024-06-19 03:18:55 +08:00
hiyouga
c0c6b8075a tiny fix
Former-commit-id: 38b6b0f52edeb8ba45aa03b415b3c0c1b0e0c1e4
2024-06-16 01:06:41 +08:00
hiyouga
2946153cea add license
Former-commit-id: d87108daa68bd40174b262be1ca65fe6e1b7ab56
2024-06-15 17:54:33 +08:00
hiyouga
8fccaf20c5 fix #4221
Former-commit-id: 6baafd4eb3147ad9f7d2952b8eb27c5486940f36
2024-06-13 02:48:21 +08:00
hoshi-hiyouga
bf3de9bfe8 Update pretrain.py
Former-commit-id: 0c292332374fb96c3fc753abde42d070a0c1dca8
2024-06-11 17:02:14 +08:00
d
da39715085 经过大量的增量预训练,进行对比试验,发现这个bug:llama3在预训练时使用的tokenizer.eos_toke是'<|end_of_text|>' ,这里在每条数据后面也得用这个,而不是'<|eot_id|>',否则很容易导致严重的性能下降
Former-commit-id: 6979f3f8480755604d8aea8164f6418126e094c5
2024-06-11 16:23:40 +08:00
mMrBun
b6d63b3324 Optimize the handling of QWEN2 in scenarios involving multiple tool calls.
Former-commit-id: 950e360ca00c29febadc14d5995de7d57b5c43a7
2024-06-10 02:00:14 +08:00
mMrBun
3f11ab800f Removed unnecessary comments.
Former-commit-id: 6ed0b0c800d416379acf8395aa852c188c107eb9
2024-06-09 18:25:22 +08:00
mMrBun
daf472994d Merge branch 'hiyouga:main' into main
Former-commit-id: 0f2609ce19492f0bab9b4880ded228b5513e5907
2024-06-09 18:17:24 +08:00
mMrBun
18a86ea104 Implemented the tool_formatter and tool_extractor for glm4 tool_format
Former-commit-id: cb1cbcb293917e960cad8f0eac7a11a122ab644a
2024-06-09 18:16:15 +08:00
hiyouga
ce40d12692 release v0.8.0
Former-commit-id: 5aa4ce47567146cd97c61623018153b41d7c1278
2024-06-08 05:20:54 +08:00
hiyouga
c6f5f69644 update data processors
Former-commit-id: ccc8b64cc21417cfd88d9b5e9bd62e15c1e834a1
2024-06-07 04:15:40 +08:00
hoshi-hiyouga
4953ded639 Merge pull request #4009 from AlongWY/main
supervised packing with greedy knapsack algorithm

Former-commit-id: 181dbb0d05655c6b46a3859cb47cfa3f86ef8abd
2024-06-07 03:48:46 +08:00
hoshi-hiyouga
e3ef239bc0 Update supervised.py
Former-commit-id: c09ad8bab38bc2f151da3a924eba225111af2481
2024-06-07 03:42:08 +08:00
hoshi-hiyouga
fd7bd911a6 Update supervised.py
Former-commit-id: 788e8232fc4ed58ab2439a9bc2e38f64e12c6eb3
2024-06-07 03:38:23 +08:00
hoshi-hiyouga
21df5f0bd0 Update supervised.py
Former-commit-id: 8cecade7082a52f413517ea20b1c5dd812db8e53
2024-06-07 03:38:04 +08:00
hiyouga
8da149ba40 rename files
Former-commit-id: 74f96efef9bcd63f65d0190c901ff9be54ccd350
2024-06-07 00:09:06 +08:00
hiyouga
e0aadd4b34 fix ppo dataset bug #4012
Former-commit-id: 149610c636bbb974e546d13fa302884ea65a6d38
2024-06-06 19:03:20 +08:00
hiyouga
94c37490d1 support glm-4
Former-commit-id: f48f5e646e2da9e02333d027033141b0e75dfcf8
2024-06-05 15:16:38 +08:00
hiyouga
0eff6a66d5 tiny fix
Former-commit-id: 5a13b3baa63225e7f79e024610722de0f87e0acc
2024-06-04 00:31:10 +08:00
hiyouga
8ecf606230 fix #3992
Former-commit-id: a18acf2abe28e37233bf8c8ed2600618ea3b62e9
2024-06-04 00:17:36 +08:00
hiyouga
64d24842fe fix data loader hint
Former-commit-id: 49b1e88e3da3be0fb78f53e5f924a9be67568a02
2024-06-03 18:28:27 +08:00
ylfeng
62d55b71a3 remove empty line
Former-commit-id: b47e3174472f458a3a8b84a66b475da8fce6db79
2024-05-31 21:43:08 +08:00
ylfeng
0feb2ad35c fix eos
Former-commit-id: 84aee579013f0c095a918a8c61611ccbb1d7fc84
2024-05-31 21:40:41 +08:00
ylfeng
8350e508d3 supervised packing with greedy knapsack algorithm
Former-commit-id: f9db439cb7511b12aa3524d5fdcc45864aebda91
2024-05-31 15:33:54 +08:00
hoshi-hiyouga
9b6bdf9449 Merge pull request #3829 from seanzhang-zhichen/add_dataset_sample_num
Add dataset sample num

Former-commit-id: 483eb47e5d670e23fb713b942f6890b8259f4363
2024-05-30 00:25:45 +08:00
hoshi-hiyouga
7b83c550ab Update loader.py
Former-commit-id: ca5dd7c6c115a359e4b50e93f4ffcc9f2955ec2f
2024-05-30 00:20:20 +08:00
hoshi-hiyouga
9fc713da89 Update loader.py
Former-commit-id: f9a88b89ca8b8f9a0c5def03b154f9d67f558edf
2024-05-30 00:17:21 +08:00
hoshi-hiyouga
c0f11a280e Update loader.py
Former-commit-id: b55fb611c57be03fb38218c7da1d96f6848496ba
2024-05-30 00:12:12 +08:00
hoshi-hiyouga
69a51cacb1 Update parser.py
Former-commit-id: 51dd454337941801d0a66eaadb0da2e007e9573d
2024-05-30 00:05:20 +08:00
hiyouga
19a3262387 fix cohere system
Former-commit-id: d0aa36b8ad02287d97930101958456c523e699d3
2024-05-29 20:58:23 +08:00
hiyouga
c05cb3769f fix #3965
Former-commit-id: 0930f5869929634baa0881167d3d6c714afc63d9
2024-05-29 20:55:51 +08:00
hiyouga
a71a6a05c3 update readme
Former-commit-id: 89ca832740731dfb121175aa5c16b13bd4944011
2024-05-29 18:39:11 +08:00
hzhaoy
ce1be3da4b add TeleChat-12B/TeleChat-12B-v2 models
Former-commit-id: 0dd632fe9e5bbf08605d4b9c6887208b7a127317
2024-05-29 15:00:37 +08:00
Yimi81
7324984127 fix yi template
Former-commit-id: dc07413e7d0b138c89eacaef17596e83ef226540
2024-05-27 13:11:25 +00:00
hiyouga
0706dbf7e6 tiny fix
Former-commit-id: c1fdf81df6ade5da7be4eb66b715f0efd171d5aa
2024-05-27 20:54:26 +08:00
hoshi-hiyouga
eceec1d7fd Update template.py
Former-commit-id: f1002b9f930758bb27794ab88a2adbe24417b076
2024-05-27 20:51:56 +08:00