79 Commits

Author SHA1 Message Date
hiyouga
db2051684b improve aligner
Former-commit-id: 7d2dc83c5e2085da6273241269c9e9d7509ae51b
2024-02-10 16:39:19 +08:00
hoshi-hiyouga
04fa6b9a3d Merge pull request #2462 from mnmueller/main
Enable Parsing of SlimOrca

Former-commit-id: 388b705a8dd6b13b20ab4d350b9b788e182e6ecd
2024-02-09 22:55:48 +08:00
hiyouga
36f092b53f improve fix tokenizer
Former-commit-id: 54ea9684ed05ac7fa5bbaa04d39636eda37cd34e
2024-02-09 14:53:14 +08:00
Mark Mueller
4bd7b8375e Slim Orca data parsing
Former-commit-id: 1d3598afa10797ba0ce30d44f52e7994587c0ce8
2024-02-08 19:32:20 +01:00
Mark Mueller
8e36c5144b Slim Orca data parsing
Former-commit-id: 6703d0546d02ea8174552ad04cb0b4bbfd7e0201
2024-02-08 17:56:18 +01:00
Mark Mueller
08967e324a Slim Orca data parsing
Former-commit-id: 7f792dfedebac8ed46eea935fed296bdb50d5f44
2024-02-08 17:54:18 +01:00
Mark Mueller
16d0bf0317 Slim Orca data parsing
Former-commit-id: 8bd41826092f0ac46ad51f91928e88e5964ae1a5
2024-02-08 17:52:36 +01:00
Mark Mueller
842b56666a SlimOrca aligner
Former-commit-id: 36d7a7596688d40524ca91f662acb391e7ce1817
2024-02-08 08:28:32 -08:00
hoshi-hiyouga
186ba72d72 Merge pull request #2423 from mayflower/main
Support for german sft and dpo

Former-commit-id: d0daaa01f9119d4feef45327bbd43964a73bc0c2
2024-02-07 15:58:20 +08:00
hiyouga
23dd337ac2 lint
Former-commit-id: 88a1bc97736bf06f292cd768fc8b61503aca1988
2024-02-07 01:10:04 +08:00
hiyouga
9debd64cef add models
Former-commit-id: 85622ae757e2ffe7f3da15f0a9123e8410d82b28
2024-02-06 14:57:23 +08:00
Johann-Peter Hartmann
e7ff724a83 Merge branch 'hiyouga:main' into main
Former-commit-id: 1ecea9de6371a78243cd4e122bdc510f83952dcc
2024-02-04 13:55:00 +00:00
hiyouga
d296e56fe9 fix #2421
Former-commit-id: 3dc86c4af9b859efbea87e47a40f666c3adb5484
2024-02-04 21:02:55 +08:00
Johann-Peter Hartmann
eb45057257 Merge branch 'hiyouga:main' into main
Former-commit-id: 63da6294ddfc5a9f3fdacb124755e89fe94d9c66
2024-02-04 12:51:25 +00:00
hiyouga
91b7fd1ec3 fix reserved label len
Former-commit-id: db0ab4d601b5e68660c00f07998ff26a8745ba90
2024-02-04 17:54:26 +08:00
hiyouga
961368a8c4 fix #2397
Former-commit-id: 51df8657348ac3153e1a44143dc61008f10ab186
2024-02-03 23:45:31 +08:00
hiyouga
a9e58740f5 fix #2376
Former-commit-id: 4ecadc35122340b3e520804270c1c1d16c696830
2024-02-03 23:14:31 +08:00
hiyouga
40514bf865 support minicpm #2404
Former-commit-id: 901faa16ccebc3fa11260daac98981dd99904af9
2024-02-03 22:36:46 +08:00
Johann-Peter Hartmann
6998b57726 add simple german chatml template chatml_de
Former-commit-id: b0ffde6e989ab59be7370cfdb6b1e3def796b2a9
2024-02-03 09:01:15 +01:00
Fallen Angel
b4c776fa93 fix eos_token_id=0 bug
when eos_token_id=0, will never add eos_token

Former-commit-id: 3399c0d6459e797946daf966e32134a12f8e48f3
2024-02-02 17:34:48 +08:00
hiyouga
509e35ffc8 fix #2282 and update tool prompt
Former-commit-id: b2fb0eca56ee835438cc20e83f36ac3f6eb95c83
2024-01-22 22:27:30 +08:00
hiyouga
9898712a24 add orion models
Former-commit-id: 6fc2d5cc0375a183ec95f003e7a27567b2a71514
2024-01-22 21:26:53 +08:00
hiyouga
48cab43cb5 add array param format
Former-commit-id: 486cc8d3600397812e3927d43ab4181f4e86f5dd
2024-01-21 22:17:48 +08:00
hiyouga
8c10530181 tiny fix
Former-commit-id: 3bc301be021bcef4dff538e5b01cf81bccb633bf
2024-01-21 13:26:12 +08:00
hiyouga
1fbf83acc2 finish agent
Former-commit-id: 3e982cc7146b408b359bb38ed4aa840183b23d72
2024-01-21 01:47:33 +08:00
hiyouga
865f48f1c3 fix api
Former-commit-id: 55f707196e772514907226d5a09fad1cdee6395d
2024-01-21 00:03:09 +08:00
hiyouga
cb8abce2f3 fix internlm2 template
Former-commit-id: a9c18255aa039550ca65145b930576edeb905d53
2024-01-20 23:33:50 +08:00
hiyouga
0be7b16955 fix cli_demo
Former-commit-id: c550987a72b0b88ea0116eb47a507142c8544df4
2024-01-20 23:27:10 +08:00
hiyouga
baaa0acdf1 fix #2260
Former-commit-id: cf818a25982c8a2808cf21f22e1e5eb4dd7e0713
2024-01-20 23:22:09 +08:00
hiyouga
4ac36c14ae release v0.5.0
Former-commit-id: 71cfdc2658e346e3fd05b3a8a678c811b3ba1bd5
2024-01-20 20:21:39 +08:00
hiyouga
b27e91222c format style
Former-commit-id: 638234ceee1b19716e45b6e5f4ea54d9122da4df
2024-01-20 20:15:56 +08:00
hiyouga
2f7684a8ee fix tests
Former-commit-id: f6d6e00337ebef8740d180836dcb18d0e6a3c59a
2024-01-20 19:58:04 +08:00
hiyouga
397eb8fbb9 fix #2249
Former-commit-id: 12043aab9c9ea2acaa3ff80232ce0451ccaa557d
2024-01-19 21:44:32 +08:00
hiyouga
a9fe47a848 fix function formatter
Former-commit-id: 35aef8b28761d787d42e548567fc6689a1502ab5
2024-01-18 16:01:07 +08:00
hiyouga
f7fcfeaa47 fix templates
Former-commit-id: a73a979afd660afd3f2091c7b813a82b3a28645d
2024-01-18 14:49:52 +08:00
hiyouga
f266d473b8 fix pretrain data loader
Former-commit-id: 5ff10fac4fd7fa0545ec7bcaa6268764f77342cf
2024-01-18 14:42:52 +08:00
hiyouga
e95c0242a8 fix dataset
Former-commit-id: 487dee066f51caea86dc65847bd326e13445de09
2024-01-18 12:59:30 +08:00
hiyouga
7f12aedc08 enable cutoff len
Former-commit-id: f1067d2b585cf24b3e48463692ac99a8222161c9
2024-01-18 12:25:42 +08:00
hiyouga
1323b8f102 add tool test
Former-commit-id: 83dbfce8c30bc1183a5c86fb77a5676bf0b753af
2024-01-18 10:26:26 +08:00
hiyouga
4e3bfb799d support function calling
Former-commit-id: d9f1cae35150cce594a7abd96dd2beb811fa33f2
2024-01-18 09:54:23 +08:00
hiyouga
21020e51ca support solar 10.7B #1907
Former-commit-id: bf73224f33c93647a6101d36bde5bf8ddfc91438
2024-01-14 00:30:30 +08:00
hiyouga
4dc16df570 fix #2147
Former-commit-id: 9aa1a2fc172730eabc4f14bcb08c0d66fcb2d844
2024-01-12 03:30:56 +08:00
hiyouga
a78cd5c5ad fix qwen template
Former-commit-id: ed216bbc4693860eaf27c2a95108f705d530cbdb
2024-01-05 16:14:56 +08:00
hiyouga
c14bf033a3 fix #2014
Former-commit-id: ce2156eaa83e2b572f4198a1eb77a835fb502319
2023-12-29 15:17:22 +08:00
hiyouga
16688b773a fix export format
Former-commit-id: e165354facf7e69f535f9b7d99438f03dbf0293d
2023-12-28 18:40:46 +08:00
hiyouga
11a6c8a9a0 fix #1909
Former-commit-id: 0bbf7118dfeb936baf03e74269211fa593003cfe
2023-12-23 14:42:20 +08:00
hiyouga
9337568e23 fix stop words
Former-commit-id: dec360d5aee58c11804b8e45dd8d4c375086887f
2023-12-20 19:06:43 +08:00
hiyouga
622d31e398 fix yi template #1895
Former-commit-id: 5af8841c4f6c97df522d2cf4e283d5ef0af21a18
2023-12-20 18:58:16 +08:00
hiyouga
c1233ab65f add max_memory for gptq #1923
Former-commit-id: c4a3977ad7278b59a9b3dadcb446bb4c99da5c9d
2023-12-20 18:15:17 +08:00
hiyouga
f64be8ee84 optimize data loading logic
Former-commit-id: ec1fe1daa98c61f62c753b22847de028b5c5cded
2023-12-20 16:15:41 +08:00