modernbert-tr-base-1k / tokenizer.json
mrbesher's picture
Fix EOS mapping to id 50000 and add [RESERVED_0-6] at 50001-50007; keep tokenizer_class=PreTrainedTokenizerFast
bdb2e56 verified
Raw History Contribute Delete
1.21 MB
File too large to display, you can check the raw version instead.