Commit Graph

144 Commits

Author SHA1 Message Date
hiyouga
f30e0a75c4 optionally replace jinja template
Former-commit-id: ba52103ba7
2024-09-25 23:02:02 +08:00
hoshi-hiyouga
1ccc2d198e flat string
Former-commit-id: f230130563
2024-09-19 16:43:42 +08:00
hoshi-hiyouga
e4d26efea9 lint
Former-commit-id: af7f547ecb
2024-09-19 16:21:43 +08:00
hoshi-hiyouga
89138b8ab8 fix bug
Former-commit-id: 98b464d2dd
2024-09-19 16:21:21 +08:00
hoshi-hiyouga
545ca7db99 improve error message
Former-commit-id: 36611d5c09
2024-09-19 16:06:00 +08:00
ybyang
daa1309466 fix: 修复function call数据集如果 function_call 值的为不合法json,异常提示且中断训练。
Former-commit-id: 953e1a0fb2
2024-09-19 15:00:10 +08:00
hiyouga
38505ae9e1 update accelerate ver for schedule_free optimizers
Former-commit-id: bdde35fd2e
2024-09-09 22:51:08 +08:00
hiyouga
2bb1ee3292 fix mm plugin
Former-commit-id: 6dc2b00fa2
2024-09-09 22:41:28 +08:00
hiyouga
c52eeb70e7 fix qwen2vl preprocess
Former-commit-id: c93795ae14
2024-09-09 22:33:33 +08:00
hiyouga
7ccb86b215 add docstrings, refactor logger
Former-commit-id: 54c6905937
2024-09-08 00:56:56 +08:00
hiyouga
995491594d tiny fix
Former-commit-id: 76f2e59504
2024-09-05 23:41:16 +08:00
hiyouga
3aa6a3e45b add e2e tests
Former-commit-id: 94d5b1bd8f
2024-09-05 21:52:28 +08:00
hoshi-hiyouga
de277a8ab8 Merge pull request #5372 from LDLINGLINGLING/main
增加了对minicpm3.0的适配'

Former-commit-id: 1274356263
2024-09-05 21:35:42 +08:00
liudan
1797fe50a4 根据代码规范修改了代码
Former-commit-id: 3d3fbaaff9
2024-09-05 20:17:55 +08:00
hiyouga
4fccc65579 support Yi-Coder models
Former-commit-id: 359ef8bb0e
2024-09-05 03:12:24 +08:00
hiyouga
9df7a26e6b video datasets
Former-commit-id: 8cafc7b055
2024-09-05 02:04:17 +08:00
liudan
09cff03026 增加了对minicpm3.0的适配'
Former-commit-id: d7ba97be48
2024-09-04 23:10:05 +08:00
hiyouga
d5ea05cfff update get template
Former-commit-id: dabad5570b
2024-09-04 22:36:20 +08:00
hoshi-hiyouga
1dfd1aaf82 Merge pull request #5323 from naem1023/feat/add-dataset-map-batch-size-argument
Add batch size of map function in the preprocessed dataset

Former-commit-id: 8f441c2b3a
2024-09-04 22:09:36 +08:00
hoshi-hiyouga
8ac74c8ccb fix #5228
Former-commit-id: 44d6947e55
2024-09-04 19:10:30 +08:00
hiyouga
a3d47818b7 fix #5344
Former-commit-id: d41d43a7c3
2024-09-04 03:06:06 +08:00
hiyouga
22deca0e9e lazy image load
Former-commit-id: 47ea97fb1b
2024-09-04 02:27:08 +08:00
hiyouga
f837ae8cb5 fix #5338
Former-commit-id: 69d0acacc3
2024-09-03 17:45:17 +08:00
naem1023
46695e42cc feat: add batch size of map function in the preprocessed dataset
Former-commit-id: 209313eeea
2024-09-02 13:52:47 +09:00
hiyouga
f13e974930 tiny fix
Former-commit-id: 3a6f19f017
2024-09-02 01:33:22 +08:00
hiyouga
859823eb23 add image num check
Former-commit-id: ce8c5a2647
2024-09-02 01:31:36 +08:00
hiyouga
bfdcc6bacf add rlhf-v dataset
Former-commit-id: 8e49940746
2024-09-01 22:57:41 +08:00
hiyouga
413a206652 fix bug
Former-commit-id: 64cb947c60
2024-09-01 21:07:49 +08:00
hiyouga
cb776752f6 fix mixed mm inputs and rlhf-v
Former-commit-id: 9967ccb3ae
2024-09-01 20:52:47 +08:00
hiyouga
09a2ecebc4 add test mm plugin
Former-commit-id: a2a8c0b92c
2024-08-31 01:53:38 +08:00
hiyouga
92c398166d tiny fix
Former-commit-id: bee1bd43b9
2024-08-30 03:21:50 +08:00
hiyouga
a83756b5e9 refactor mm training
Former-commit-id: 3382317e32
2024-08-30 02:14:31 +08:00
hoshi-hiyouga
5730243179 fix bug
Former-commit-id: a8f22d8895
2024-08-30 02:05:26 +08:00
simonJJJ
d9fd8f26cc update
Former-commit-id: 734e019cc1
2024-08-28 20:22:46 +08:00
simonJJJ
8a09b1e732 initial-commit
Former-commit-id: aeb85f200b
2024-08-28 16:51:35 +08:00
hoshi-hiyouga
a7604a95c1 Merge pull request #5156 from YeQiuO/main
fix Llama-template's system prompt bug

Former-commit-id: 15be296347
2024-08-20 00:09:03 +08:00
hoshi-hiyouga
103132aa99 Update template.py
Former-commit-id: ec72eeca52
2024-08-20 00:03:33 +08:00
hoshi-hiyouga
a921505f59 Update template.py
Former-commit-id: 5f3300ec5d
2024-08-19 23:40:16 +08:00
Huiyu Chen
66a7f4f128 Add SailorLLM template
Former-commit-id: 2502833a77
2024-08-15 15:10:14 +08:00
“Wzw”
3e159a0a83 fix Llama-template's system prompt bug
Former-commit-id: bcbbf45063
2024-08-12 19:22:12 +08:00
hiyouga
b5146facff follow #5115
Former-commit-id: c87023d539
2024-08-09 18:03:00 +08:00
hoshi-hiyouga
397e4daa5d Merge pull request #5115 from YeQiuO/main
fix: `Train on the last turn only` truncate bug
Former-commit-id: 51542cb15f
2024-08-09 17:58:27 +08:00
hoshi-hiyouga
54f57fb354 Update template.py
Former-commit-id: 4f62e1cb24
2024-08-09 16:27:42 +08:00
“Wzw”
13e5fff97a mask_history args verify valid
Former-commit-id: 2fa1e0b2ad
2024-08-08 10:12:01 +08:00
“Wzw”
0bd25c3a6b fix mask_history tiny bug
Former-commit-id: b5ca86cc07
2024-08-08 10:09:33 +08:00
moontidef
733cb9087b fix: fix the deepseekcoder template to avoid repeat problem
Former-commit-id: b82ecbedd0
2024-08-05 23:55:45 +08:00
hoshi-hiyouga
0ac7824822 Merge pull request #4892 from piamo/main
update deepseek template

Former-commit-id: 8a2846cfe1
2024-07-26 11:49:34 +08:00
hiyouga
d63beb7a24 fix #4928
Former-commit-id: 091010492b
2024-07-24 17:00:29 +08:00
hiyouga
8d06679a3f fix flashattn + packing
Former-commit-id: 4135e69406
2024-07-21 17:07:45 +08:00
huangpan.foo
ee4c3f32d1 update deepseek template
Former-commit-id: 44e48e2b82
2024-07-19 15:02:54 +08:00