hiyouga
88d9f47a0b
fix #3116
...
Former-commit-id: ce77d98872
2024-04-03 14:47:59 +08:00
hiyouga
bf5ffeeae0
simplify readme
...
Former-commit-id: 92dab8a90b
2024-04-02 20:07:43 +08:00
hiyouga
f4be51f356
add moe aux loss control #3085
...
Former-commit-id: b267aeb53f
2024-04-02 14:26:31 +08:00
hiyouga
c7104f8fab
fix #3022
...
Former-commit-id: 9ddbe2866a
2024-04-02 13:58:39 +08:00
hiyouga
ab6476a9a4
set dev version
...
Former-commit-id: dd73a0c248
2024-04-01 23:24:08 +08:00
hiyouga
829cf6458a
fix #3083
...
Former-commit-id: 4a6ca621c0
2024-04-01 22:53:52 +08:00
hiyouga
8d987b7af7
add qwen1.5 moe
...
Former-commit-id: 54b7d34908
2024-04-01 21:49:40 +08:00
hiyouga
34f1de0574
fix #3077
...
Former-commit-id: aee634cd20
2024-04-01 21:35:18 +08:00
hiyouga
b7468ea0a8
support infer 4bit model on GPUs #3023
...
Former-commit-id: eb259cc573
2024-04-01 17:34:04 +08:00
hiyouga
cefe7f7bcf
update webui
...
Former-commit-id: d0842f6828
2024-04-01 16:23:28 +08:00
hiyouga
bd52e2b404
fix ORPO loss
...
Former-commit-id: 816d714146
2024-04-01 14:42:41 +08:00
hiyouga
69e1d39832
fix IPO and ORPO loss
...
Former-commit-id: 5b9b40403d
2024-04-01 14:37:53 +08:00
hiyouga
e7ade84bba
fix plots
...
Former-commit-id: 5907216a1c
2024-03-31 19:43:48 +08:00
hiyouga
b873dcb09d
use log1p in orpo loss
...
https://github.com/huggingface/trl/pull/1491
Former-commit-id: 68aaa4904b
2024-03-31 19:27:08 +08:00
hiyouga
ddad9be81d
update readme
...
Former-commit-id: 099db6acc0
2024-03-31 18:46:34 +08:00
hiyouga
35b3516812
support orpo in webui
...
Former-commit-id: 5195add324
2024-03-31 18:34:59 +08:00
hiyouga
2f878bde11
support ORPO
...
Former-commit-id: 17bf8a2c3a
2024-03-31 18:29:50 +08:00
hiyouga
3cf35e57db
tiny fix
...
Former-commit-id: 27776c3474
2024-03-31 00:10:29 +08:00
marko1616
5721074af1
fix blank line contains whitespace
...
Former-commit-id: d9a5134617
2024-03-30 23:46:55 +08:00
marko1616
67c05c2031
Fix Llama model save for full param train
...
Former-commit-id: eb178eaff3
2024-03-30 23:45:04 +08:00
hiyouga
3bf6dde3a5
support save args in webui #2807 #3046
...
some ideas are borrowed from @marko1616
Former-commit-id: 7a086ed333
2024-03-30 23:09:12 +08:00
hiyouga
808ad2071f
upgrade gradio to 4.21.0
...
Former-commit-id: 831c5321ac
2024-03-30 20:37:08 +08:00
hiyouga
fc066cad7f
release v0.6.1
...
Former-commit-id: ca793028c6
2024-03-29 11:36:08 +08:00
hiyouga
e4f3d583df
fix #2982
...
Former-commit-id: 8d603f8820
2024-03-28 20:22:31 +08:00
hiyouga
eac2a5b1d3
fix #3010
...
Former-commit-id: b19c14870d
2024-03-28 18:31:17 +08:00
hiyouga
89c400633a
update trainers
...
Former-commit-id: 8c77b10912
2024-03-28 18:16:27 +08:00
hoshi-hiyouga
ae9ad13f2a
fix ds optimizer
...
Former-commit-id: 3bcd41b639
2024-03-26 23:39:56 +08:00
hiyouga
c311375b50
fix bug
...
Former-commit-id: 3164b4f11b
2024-03-26 17:30:12 +08:00
hiyouga
ec94e5e876
fix #2961
...
Former-commit-id: 511f675402
2024-03-26 17:26:14 +08:00
hiyouga
62312716d9
release v0.6.0 (real)
...
Former-commit-id: ba70aca8fb
2024-03-25 23:37:48 +08:00
hiyouga
196a33cca4
tiny fix
...
Former-commit-id: 98a42cbdaa
2024-03-25 23:28:52 +08:00
hiyouga
b18749fb01
add arg check
...
Former-commit-id: 1484f76a95
2024-03-25 22:42:58 +08:00
hiyouga
27151b8c65
release v0.6.0
...
Former-commit-id: 6f2b563f12
2024-03-25 22:38:56 +08:00
hiyouga
2d73831177
tiny fix
...
Former-commit-id: 558a538724
2024-03-25 21:18:08 +08:00
marko1616
1d0e24549f
pass ruff check
...
Former-commit-id: c8f0d99704
2024-03-24 16:12:10 +08:00
marko1616
a68101cbbb
fix Llama lora merge crash
...
Former-commit-id: 6f080fdba3
2024-03-24 03:06:11 +08:00
marko1616
645c27e5e2
fix Llama lora merge crash
...
Former-commit-id: 51349ea1cc
2024-03-24 02:55:23 +08:00
marko1616
c083708433
fix Llama lora merge crash
...
Former-commit-id: c1e2c4ea45
2024-03-24 02:44:35 +08:00
hiyouga
84c3d509fa
fix #2936
...
Former-commit-id: 140ad4ad56
2024-03-24 00:43:21 +08:00
hiyouga
75829c8699
fix #2928
...
Former-commit-id: 7afbc85dae
2024-03-24 00:34:54 +08:00
hiyouga
58aa576ae5
fix #2941
...
Former-commit-id: a1c8c98c5f
2024-03-24 00:28:44 +08:00
hiyouga
7999836fb6
support fsdp + qlora
...
Former-commit-id: 8408225162
2024-03-21 00:36:06 +08:00
hiyouga
8717e98200
fix #2777 #2895
...
Former-commit-id: 9bec3c98a2
2024-03-20 17:59:45 +08:00
hiyouga
cf149bf43c
fix #2346
...
Former-commit-id: 7b8f502901
2024-03-20 17:56:33 +08:00
hiyouga
3d483e0914
fix packages
...
Former-commit-id: 8e04794b2d
2024-03-17 22:32:03 +08:00
hiyouga
a5537f3ee8
fix patcher
...
Former-commit-id: 85c376fc1e
2024-03-15 19:18:42 +08:00
hoshi-hiyouga
30765baa91
Merge pull request #2849 from S3Studio/DockerizeSupport
...
Improve Dockerize support
Former-commit-id: 113cc04719
2024-03-15 19:16:02 +08:00
hiyouga
06860e8f0f
fix export
...
Former-commit-id: 6bc2c23b6d
2024-03-15 15:06:30 +08:00
S3Studio
46ef7416e6
Use official Nvidia base image
...
Note that the flash-attn library is installed in this image and the qwen model will use it automatically.
However, if the the host machine's GPU is not compatible with the library, an exception will be raised during the training process as follows:
FlashAttention only supports Ampere GPUs or newer.
So if the --flash_attn flag is not set, an additional patch for the qwen model's config is necessary to set the default value of use_flash_attn from "auto" to False.
Former-commit-id: e75407febd
2024-03-15 08:59:13 +08:00
hiyouga
7ef49586be
tiny fix
...
Former-commit-id: 6ebde4f23e
2024-03-14 21:19:06 +08:00
