hiyouga
|
5acaa476d6
|
update hparams
Former-commit-id: 1c4feac44192b1f540208837f5a530b0d3f5fb37
|
2024-07-03 23:18:58 +08:00 |
|
ancv
|
20fdf177e8
|
move efficient_packing from data_args to model_args
Former-commit-id: 7b61659c707480bcf8c802c73e10d12ad5b9b965
|
2024-07-02 18:37:55 +07:00 |
|
hoshi-hiyouga
|
a715490c2a
|
Merge branch 'main' into main
Former-commit-id: 7be442f37d53a0c6324728fa1fa8e2c84d7f0fa5
|
2024-07-01 21:01:09 +08:00 |
|
hiyouga
|
67d2eb6b2a
|
fix #4402 #4617
Deprecate reserved_label_len arg
Former-commit-id: 4b6568984c0be4b31e7aa91b7c0d52b7f7b12b0b
|
2024-07-01 01:19:27 +08:00 |
|
hiyouga
|
05f3a3c944
|
tiny fix
Former-commit-id: f7f440986b0ae3b38ea9f2da80789629d4f79ea1
|
2024-06-16 01:06:41 +08:00 |
|
ancv
|
f91fe10985
|
remove some unused params
Former-commit-id: fef8132c50505a5fb6a246bd024491bd31798a3c
|
2024-06-15 23:00:55 +07:00 |
|
hiyouga
|
bb88536166
|
add license
Former-commit-id: 69cfc98d7c81756a5ab6bf962240e393e449fef0
|
2024-06-15 17:54:33 +08:00 |
|
hiyouga
|
49b58fd6af
|
fix #4221
Former-commit-id: 05a3be4853b941909e7d193c31e8d62c8c5f879b
|
2024-06-13 02:48:21 +08:00 |
|
ancv
|
c7ab302c69
|
implement efficient packing without cross-contamination attention
Former-commit-id: a64a5305c0da5ef092d4cc26faf829bb44de65d1
|
2024-06-12 11:56:01 +07:00 |
|
hoshi-hiyouga
|
cc9717e2f2
|
Update pretrain.py
Former-commit-id: e2317b2a84149e39fddfd6366be3de23dfb71f82
|
2024-06-11 17:02:14 +08:00 |
|
d
|
77bf3d66c7
|
经过大量的增量预训练，进行对比试验，发现这个bug：llama3在预训练时使用的tokenizer.eos_toke是'<|end_of_text|>' ，这里在每条数据后面也得用这个，而不是'<|eot_id|>'，否则很容易导致严重的性能下降
Former-commit-id: ef470561f742b16eaa0f99c4cadecd7c84ce6bd2
|
2024-06-11 16:23:40 +08:00 |
|
hiyouga
|
8c4c2e580c
|
update data processors
Former-commit-id: 04b138cbcb8b9a72e4bbda6c65843bb459e525e7
|
2024-06-07 04:15:40 +08:00 |
|
hoshi-hiyouga
|
07f33e7641
|
Merge pull request #4009 from AlongWY/main
supervised packing with greedy knapsack algorithm
Former-commit-id: 5ded166b39a75a98ded5733678f5a1eab7d4cc71
|
2024-06-07 03:48:46 +08:00 |
|
hoshi-hiyouga
|
1998c641af
|
Update supervised.py
Former-commit-id: 04b6c2a754e602e0b698cfe6c255c2f2486d8865
|
2024-06-07 03:42:08 +08:00 |
|
hoshi-hiyouga
|
be1e5f9d62
|
Update supervised.py
Former-commit-id: 49993c4f4e1f871a22ff0196afe60026b668a4dc
|
2024-06-07 03:38:23 +08:00 |
|
hoshi-hiyouga
|
fdeec6db52
|
Update supervised.py
Former-commit-id: 67625b5278a839c12a3e4245f9e90af67d8b11b4
|
2024-06-07 03:38:04 +08:00 |
|
hiyouga
|
fcb134e144
|
rename files
Former-commit-id: e1a8431770fc36c0c9ee7fed4abbc3d7fdcc5efd
|
2024-06-07 00:09:06 +08:00 |
|
hiyouga
|
ca95e98ca0
|
fix ppo dataset bug #4012
Former-commit-id: 7fc51b2e93698ae5e012566af8481f4d861c873d
|
2024-06-06 19:03:20 +08:00 |
|
ylfeng
|
b1b7d735b3
|
remove empty line
Former-commit-id: 3164710971a6d6545629f5bf133f98de5ff0991a
|
2024-05-31 21:43:08 +08:00 |
|
ylfeng
|
230c69f7ce
|
fix eos
Former-commit-id: 6e236c952958cbfe50b5dcb7b8eff6aea8477922
|
2024-05-31 21:40:41 +08:00 |
|
ylfeng
|
bfc43558ef
|
supervised packing with greedy knapsack algorithm
Former-commit-id: 24d12396c9aabd49da0b08719068f24679111cc6
|
2024-05-31 15:33:54 +08:00 |
|
hiyouga
|
97a23e1cbe
|
update readme
Former-commit-id: b8d0170fe0d094acce85dcb5f91775e4685ee055
|
2024-05-27 18:14:02 +08:00 |
|
hiyouga
|
bf59383783
|
refactor data preprocessing, fix mllm rlhf
Former-commit-id: 53ff2dd24f9121ea30c95063bb72e49a9b31e980
|
2024-05-24 04:08:25 +08:00 |
|