improve fix tokenizer

Former-commit-id: 54ea9684ed
This commit is contained in:
hiyouga
2024-02-09 14:53:14 +08:00
parent 186ba72d72
commit 36f092b53f
7 changed files with 105 additions and 71 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 137 KiB

After

Width:  |  Height:  |  Size: 142 KiB