hoshi-hiyouga 205aca5b03 Merge pull request #6078 from wtmlon/support-efficient-tokens-calculation
support effective tokens calculation on sft/dpo

Former-commit-id: d0510e6d49b43c5ffadd8af653c3bdecc1582417
2024-11-20 13:43:15 +08:00
..
2024-11-02 20:59:01 +08:00
2024-11-02 18:31:04 +08:00
2024-11-02 20:59:01 +08:00
2024-11-02 18:31:04 +08:00
2024-11-19 20:33:18 +08:00
2024-05-16 18:39:08 +08:00
2024-11-02 20:59:01 +08:00
2024-11-02 12:41:44 +08:00
2024-11-02 18:31:04 +08:00
2024-11-02 18:31:04 +08:00