modernbert-tr-base-1k / tokenizer.json

Commit History

  1. Fix EOS mapping to id 50000 and add [RESERVED_0-6] at 50001-50007; keep tokenizer_class=PreTrainedTokenizerFast

    Verified signature
    bdb2e56
  2. Add [RESERVED_0-7] tokens at ids 50000-50007; keep tokenizer_class=PreTrainedTokenizerFast

    Verified signature
    0cbc42d

  1. Fix: [EOS] at 50000, add [RESERVED_0-6] at 50001-50007, vocab_size=50008

    Verified signature
    6d05e64
  2. Fix: restore [EOS] at 50000, add reserved tokens to match embedding size 50008

    Verified signature
    49216b8
  3. Add 7 reserved special tokens to match embedding size (50008)

    Verified signature
    3e90da5

  1. Upload 6 files

    Verified signature
    8e242a3